toRecords fails to convert pl.Date values

Have you tried latest version of polars?

yes

What version of polars are you using?

0.7.2

What operating system are you using polars on?

Debian 12

What node version are you using

node 18.13.0

Describe your bug.

With a pl.Date column in the data frame, using the toRecords function produces objects which have dates at the unix epoch.

What are the steps to reproduce the behavior?
import pl from "nodejs-polars";

let df = pl.DataFrame({
  date: [new Date()],
});

df = df.withColumn(pl.col("date").cast(pl.Date).alias("date"));

console.log(df.toString());
console.log(df.toRecords());

What is the actual behavior?
shape: (1, 1)
┌────────────┐
│ date       │
│ ---        │
│ date       │
╞════════════╡
│ 2023-03-09 │
└────────────┘

{date: Thu Jan 01 1970 01:00:19 GMT+0100 (Greenwich Mean Time)}

What is the expected behavior?
shape: (1, 1)
┌────────────┐
│ date       │
│ ---        │
│ date       │
╞════════════╡
│ 2023-03-09 │
└────────────┘

{date: Thu Mar 09 2023 00:00:00 GMT+0100 (Greenwich Mean Time)}

This works in 0.8.0:
let df = pl.DataFrame({
  date: [new Date()],
});

df = df.withColumn(pl.col("date").date.strftime("%Y-%m-%d").alias("date"));

console.log(df.toString());
console.log(df.toRecords());

shape: (1, 1)
┌────────────┐
│ date       │
│ ---        │
│ str        │
╞════════════╡
│ 2023-06-18 │
└────────────┘
[ { date: '2023-06-18' } ]

I think you have misunderstood, as your example is different since it's just converting a pl.Datetime column into strings.
I'm interested in storing pl.Date type in the dataframe, which should map to Date objects in the records form (pl.Datetime works correctly and can be used as a workaround, but I don't need that precision, just the date).

@universalmind303 Something is happening during col.get(idx); on this line
Would you be able to shade some light? Thx

The code above works fine in 0.8.2, please re-open is this still an issue. Thx


Optimized compiled Tensorflow uses almost 2x more memory than non-optimized binary

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): Both
TensorFlow version (use command below): ('v1.0.0-65-g4763edf-dirty', '1.0.1') (compiled at the same git commit of the binary version)
Bazel version (if compiling from source): 0.4.5
CUDA/cuDNN version: N/A (CPU only)
GPU model and memory: N/A
Exact command to reproduce:

import pdb
import numpy as np
import tensorflow as tf

def get_layer(input_size, output_size, name):
    # W_val is loaded from a file using numpy.load
    W_val = np.random.normal(scale=0.1, size=(input_size, output_size)).astype(np.float32)
    W = tf.get_variable(name='W_{}'.format(name), shape=(input_size, output_size),
                        initializer=tf.constant_initializer(value=W_val, dtype=tf.float32),
                        dtype=tf.float32)
    b = tf.get_variable(name='b_{}'.format(name), shape=(output_size,),
                        initializer=tf.constant_initializer(value=0.1, dtype=tf.float32),
                        dtype=tf.float32)
    return W, b

sessions = []
for i in range(3):
    g = tf.Graph()
    with g.as_default():
        W1, b1 = get_layer(158238, 900, '1')
        W2, b2 = get_layer(900, 1000, '2')
        W3, b3 = get_layer(1000, 1, '3')

        init = tf.global_variables_initializer()
    session = tf.Session(graph=g)
    session.run(init)
    print 'Loaded {}'.format(i)
    sessions.append(session)

pdb.set_trace()

Describe the problem
After running the code snippet under the non-optimized binary Tensorflow installation, the used memory is ~6GB. However, when the same snippet is run with Tensorflow compiled with -c opt --copt=-march=native directives, the memory usage is ~11GB (1.8x larger).
Note that there's no memory-usage difference when I use tf.truncated_normal(stddev=0.1...) instead of tf.constant_initializer with a numpy array.
Not sure if this is a bug, or a side-effect of the optimized version or if there's something I can do to optimize memory usage?
Source code / logs
I can attach chrome traces if necessary.

@tfboyd is there anyone who could take a look?

Could you try to disable jemalloc?

Which CPU are you using?  If we repro, I would like to compile with the same optimizations.  Thank you.  I ask because if I repo with AVX but the issue is with AVX2 for example I would not see it.

Haven't tried with jemalloc disabled yet but to answer the question about CPU it's an Amazon ec2 instance with Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz.

Just re-compiled with TF_NEED_JEMALLOC environment variable set to 0, memory usage is the same (i.e. equal to the compiled version with jemalloc enabled).

What if you run with tcmalloc?
sudo apt-get install google-perftools
export LD_PRELOAD="/usr/lib/libtcmalloc.so.4"
On Mon, May 8, 2017 at 7:53 AM, Fayez<EMAIL_ADDRESS>wrote:

Just re-compiled with TF_NEED_JEMALLOC environment variable set to 0,
memory usage is the same (i.e. equal to the compiled version with jemalloc
enabled).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/9742#issuecomment-299889130,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AYoxVP-OdUbvKvzohzA4BcwrxC0AgK-mks5r3yxmgaJpZM4NTSLO
.

Tested with tcmalloc on both jemalloc enabled and disabled versions, memory usage is the same as well. Here's the output log with tcmalloc:
~# LD_PRELOAD="/usr/lib/libtcmalloc.so.4" python test.py
tcmalloc: large alloc<PHONE_NUMBER> bytes == 0x3488000 @  0x7f268db1fd1b 0x7f268be9835c 0x7f268becfe73 0x7f268bed2b19 0x7f268bf4b1e8 0x7f263b52ece5 0x7f263b571c00 0x4cada2 0x4c9d8f 0x4c2765 0x4c2509 0x4f1def 0x4ec652 0x4eae31 0x49e14a 0x7f268d527830 0x49d9d9 (nil)
Loaded 0
Loaded 1
Loaded 2

Great! Could you follow the pprof instructions to understand what does the big allocations?

I ran the profiler tool against the pip version and the compiled version. First, here's the printed messages for each:
Pip version
~# LD_PRELOAD="/usr/lib/libtcmalloc.so.4" HEAPPROFILE=/root/tf_profile python test.py
Starting tracking the heap
Starting tracking the heap
tcmalloc: large alloc<PHONE_NUMBER> bytes == 0x4326000 @  0x7fa0319cfd1b 0x7fa02fa7835c 0x7fa02faafe73 0x7fa02fab2b19 0x7fa02fb2b1e8 0x7f9fdef7ece5 0x7f9fdefc1c00 0x4cada2 0x4c9d8f 0x4c2765 0x4c2509 0x4f1def 0x4ec652 0x4eae31 0x49e14a 0x7fa0313d7830 0x49d9d9 (nil)
Dumping heap profile to /root/tf_profile.0001.heap (1387 MB allocated cumulatively, 1105 MB currently in use)
Dumping heap profile to /root/tf_profile.0002.heap (1649 MB currently in use)
Dumping heap profile to /root/tf_profile.0003.heap (3016 MB allocated cumulatively, 1649 MB currently in use)
Dumping heap profile to /root/tf_profile.0004.heap (2192 MB currently in use)
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Dumping heap profile to /root/tf_profile.0005.heap (4676 MB allocated cumulatively, 1657 MB currently in use)
Dumping heap profile to /root/tf_profile.0006.heap (5769 MB allocated cumulatively, 1657 MB currently in use)
Dumping heap profile to /root/tf_profile.0007.heap (Exiting, 561 MB in use)

Compiled version
~# LD_PRELOAD="/usr/lib/libtcmalloc.so.4" HEAPPROFILE=/root/tf_profile python test.py
Starting tracking the heap
Starting tracking the heap
tcmalloc: large alloc<PHONE_NUMBER> bytes == 0x3c70000 @  0x7f4c16a35d1b 0x7f4c14ade35c 0x7f4c14b15e73 0x7f4c14b18b19 0x7f4c14b911e8 0x7f4bc3fd4ce5 0x7f4bc4017c00 0x4cada2 0x4c9d8f 0x4c2765 0x4c2509 0x4f1def 0x4ec652 0x4eae31 0x49e14a 0x7f4c1643d830 0x49d9d9 (nil)
Dumping heap profile to /root/tf_profile.0001.heap (1390 MB allocated cumulatively, 1105 MB currently in use)
Dumping heap profile to /root/tf_profile.0002.heap (1649 MB currently in use)
Dumping heap profile to /root/tf_profile.0003.heap (3020 MB allocated cumulatively, 1649 MB currently in use)
Dumping heap profile to /root/tf_profile.0004.heap (2192 MB currently in use)
Dumping heap profile to /root/tf_profile.0005.heap (4680 MB allocated cumulatively, 1657 MB currently in use)
Dumping heap profile to /root/tf_profile.0006.heap (5777 MB allocated cumulatively, 2754 MB currently in use)
Dumping heap profile to /root/tf_profile.0007.heap (6867 MB allocated cumulatively, 2751 MB currently in use)
Dumping heap profile to /root/tf_profile.0008.heap (7960 MB allocated cumulatively, 2751 MB currently in use)
Dumping heap profile to /root/tf_profile.0009.heap (Exiting, 561 MB in use)

Up until heap 4 in both cases, memory usage is the same and both graphs (generated by google-pprof tool) are exactly the same based on a quick look.
At heap 5, the memory usage is still the same but in the compiled version's case I see some parallelization in the graph where there are 2 nodes with ~540MB each instead of 1 node with 1090MB in the pip version's case.
Heap 6 is where the memory usage doubles in the compiled version's case. In the pip version there's 1 big unnamed node with 1637.6MB. However, in the compiled version there are 3 nodes with ~500MB each (initcmath + 2 other unnamed nodes) + a fourth large node called tensorflow PartialRunSetupRequest MergePartialFromCodedStream with 1093.8MB usage. I didn't see this particular node in any of the heap files of the pip-version. I can attach a screenshot but it becomes unreadable when I zoom out enough to view the whole thing. Let me know if you'd like me to attach the heap files.
In heap 7 of the compiled version, that same node appears with double the memory as well (2183.7MB).
I used a simplified version of my original snippet to generate the heap files:
import numpy as np
import tensorflow as tf

def get_layer(input_size, output_size, name):
    # supposedly loaded from a saved file
    W_val = np.random.normal(scale=0.1, size=(input_size, output_size)).astype(np.float32)
    W = tf.get_variable(name='W_{}'.format(name), shape=(input_size, output_size),
                        initializer=tf.constant_initializer(value=W_val, dtype=tf.float32),
                        dtype=tf.float32)
    b = tf.get_variable(name='b_{}'.format(name), shape=(output_size,),
                        initializer=tf.constant_initializer(value=0.1, dtype=tf.float32),
                        dtype=tf.float32)
    return W, b

g = tf.Graph()
with g.as_default():
    W1, b1 = get_layer(158238, 900, '1')
    W2, b2 = get_layer(900, 1000, '2')
    W3, b3 = get_layer(1000, 1, '3')
    init = tf.global_variables_initializer()
session = tf.Session(graph=g)
session.run(init)
session.close()

Could it be that one of the version is using Python implementation of
protobuf instead of cpp?
Try this
python -c "from google.protobuf.internal import api_implementation;
print(api_implementation._default_implementation_type)"
It should print "cpp"
On Mon, May 8, 2017 at 12:30 PM, Fayez<EMAIL_ADDRESS>wrote:

I ran the profiler tool against the pip version and the compiled version.
First, here's the printed messages for each:
Pip version
~# LD_PRELOAD="/usr/lib/libtcmalloc.so.4" HEAPPROFILE=/root/tf_profile python test.py
Starting tracking the heap
Starting tracking the heap
tcmalloc: large alloc<PHONE_NUMBER> bytes == 0x4326000 @  0x7fa0319cfd1b 0x7fa02fa7835c 0x7fa02faafe73 0x7fa02fab2b19 0x7fa02fb2b1e8 0x7f9fdef7ece5 0x7f9fdefc1c00 0x4cada2 0x4c9d8f 0x4c2765 0x4c2509 0x4f1def 0x4ec652 0x4eae31 0x49e14a 0x7fa0313d7830 0x49d9d9 (nil)
Dumping heap profile to /root/tf_profile.0001.heap (1387 MB allocated cumulatively, 1105 MB currently in use)
Dumping heap profile to /root/tf_profile.0002.heap (1649 MB currently in use)
Dumping heap profile to /root/tf_profile.0003.heap (3016 MB allocated cumulatively, 1649 MB currently in use)
Dumping heap profile to /root/tf_profile.0004.heap (2192 MB currently in use)
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Dumping heap profile to /root/tf_profile.0005.heap (4676 MB allocated cumulatively, 1657 MB currently in use)
Dumping heap profile to /root/tf_profile.0006.heap (5769 MB allocated cumulatively, 1657 MB currently in use)
Dumping heap profile to /root/tf_profile.0007.heap (Exiting, 561 MB in use)
Compiled version
~# LD_PRELOAD="/usr/lib/libtcmalloc.so.4" HEAPPROFILE=/root/tf_profile python test.py
Starting tracking the heap
Starting tracking the heap
tcmalloc: large alloc<PHONE_NUMBER> bytes == 0x3c70000 @  0x7f4c16a35d1b 0x7f4c14ade35c 0x7f4c14b15e73 0x7f4c14b18b19 0x7f4c14b911e8 0x7f4bc3fd4ce5 0x7f4bc4017c00 0x4cada2 0x4c9d8f 0x4c2765 0x4c2509 0x4f1def 0x4ec652 0x4eae31 0x49e14a 0x7f4c1643d830 0x49d9d9 (nil)
Dumping heap profile to /root/tf_profile.0001.heap (1390 MB allocated cumulatively, 1105 MB currently in use)
Dumping heap profile to /root/tf_profile.0002.heap (1649 MB currently in use)
Dumping heap profile to /root/tf_profile.0003.heap (3020 MB allocated cumulatively, 1649 MB currently in use)
Dumping heap profile to /root/tf_profile.0004.heap (2192 MB currently in use)
Dumping heap profile to /root/tf_profile.0005.heap (4680 MB allocated cumulatively, 1657 MB currently in use)
Dumping heap profile to /root/tf_profile.0006.heap (5777 MB allocated cumulatively, 2754 MB currently in use)
Dumping heap profile to /root/tf_profile.0007.heap (6867 MB allocated cumulatively, 2751 MB currently in use)
Dumping heap profile to /root/tf_profile.0008.heap (7960 MB allocated cumulatively, 2751 MB currently in use)
Dumping heap profile to /root/tf_profile.0009.heap (Exiting, 561 MB in use)
Up until heap 4 in both cases, memory usage is the same and both graphs
(generated by google-pprof tool) are exactly the same based on a quick
look.
At heap 5, the memory usage is still the same but in the compiled
version's case I see some parallelization in the graph where there are 2
nodes with ~540MB each instead of 1 node with 1090MB in the pip version's
case.
Heap 6 is where the memory usage doubles in the compiled version's case.
In the pip version there's 1 big unnamed node with 1637.6MB. However, in
the compiled version there are 3 nodes with ~500MB each (initcmath + 2
other unnamed nodes) + a fourth large node called tensorflow
PartialRunSetupRequest MergePartialFromCodedStream with 1093.8MB usage. I
didn't see this particular node in any of the heap files of the
pip-version. I can attach a screenshot but it becomes unreadable when I
zoom out enough to view the whole thing. Let me know if you'd like me to
attach the heap files.
In heap 7 of the compiled version, that same node appears with double the
memory as well (2183.7MB).
I used a simplified version of my original snippet to generate the heap
files:
import numpy as npimport tensorflow as tf
def get_layer(input_size, output_size, name):
# supposedly loaded from a saved file
W_val = np.random.normal(scale=0.1, size=(input_size, output_size)).astype(np.float32)
W = tf.get_variable(name='W_{}'.format(name), shape=(input_size, output_size),
initializer=tf.constant_initializer(value=W_val, dtype=tf.float32),
dtype=tf.float32)
b = tf.get_variable(name='b_{}'.format(name), shape=(output_size,),
initializer=tf.constant_initializer(value=0.1, dtype=tf.float32),
dtype=tf.float32)
return W, b
g = tf.Graph()with g.as_default():
W1, b1 = get_layer(158238, 900, '1')
W2, b2 = get_layer(900, 1000, '2')
W3, b3 = get_layer(1000, 1, '3')
init = tf.global_variables_initializer()
session = tf.Session(graph=g)
session.run(init)
session.close()
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/9742#issuecomment-299966164,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AYoxVHu9SJzrRAiqslcLQwpRRaD2Ku51ks5r321OgaJpZM4NTSLO
.

I ran the command in both environments (compiled and pip version - Docker containers) and the output is "cpp"

After seeing #9823 (not sure if related to this), I tried the memory_util script suggested by @yaroslavvb but the output is pretty much the same for both environments:
Binary
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 32
I tensorflow/core/common_runtime/direct_session.cc:83] Direct session inter op parallelism threads: 32
        0                Unknown (from Proto)(1-cpu)   569656800   569656800 cpu
        1                Unknown (from Proto)(1-cpu)  -569656800           0 cpu
        2               W_1/Initializer/Const(2-cpu)   569656800   569656800 cpu
        3                Unknown (from Proto)(3-cpu)        3600   569660400 cpu
        4                Unknown (from Proto)(3-cpu)       -3600   569656800 cpu
        5               b_1/Initializer/Const(4-cpu)        3600   569660400 cpu
        6                Unknown (from Proto)(5-cpu)     3600000   573260400 cpu
        7                Unknown (from Proto)(5-cpu)    -3600000   569660400 cpu
        8               W_2/Initializer/Const(6-cpu)     3600000   573260400 cpu
        9                Unknown (from Proto)(7-cpu)        4000   573264400 cpu
       10                Unknown (from Proto)(7-cpu)       -4000   573260400 cpu
       11               b_2/Initializer/Const(8-cpu)        4000   573264400 cpu
       12                Unknown (from Proto)(9-cpu)        4000   573268400 cpu
       13                Unknown (from Proto)(9-cpu)       -4000   573264400 cpu
       14              W_3/Initializer/Const(10-cpu)        4000   573268400 cpu
       25               Unknown (from Proto)(13-cpu)   569656800<PHONE_NUMBER> cpu
       26               Unknown (from Proto)(13-cpu)  -569656800   573268400 cpu
       28               Unknown (from Proto)(14-cpu)        3600   573272000 cpu
       29               Unknown (from Proto)(14-cpu)       -3600   573268400 cpu
       31                         b_1/Assign(16-cpu)        3600   573272000 cpu
       32                         W_1/Assign(17-cpu)   569656800<PHONE_NUMBER> cpu
       34               Unknown (from Proto)(15-cpu)     3600000<PHONE_NUMBER> cpu
       35               Unknown (from Proto)(15-cpu)    -3600000<PHONE_NUMBER> cpu
       37               Unknown (from Proto)(18-cpu)        4000<PHONE_NUMBER> cpu
       38               Unknown (from Proto)(18-cpu)       -4000<PHONE_NUMBER> cpu
       40               Unknown (from Proto)(19-cpu)        4000<PHONE_NUMBER> cpu
       41               Unknown (from Proto)(19-cpu)       -4000<PHONE_NUMBER> cpu
       48                         W_2/Assign(22-cpu)     3600000<PHONE_NUMBER> cpu
       49                         b_2/Assign(23-cpu)        4000<PHONE_NUMBER> cpu
       52                         W_3/Assign(24-cpu)        4000<PHONE_NUMBER> cpu
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 2 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 4 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 6 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 8 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 10 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 12 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 17 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 16 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 23 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 21 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 22 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 24 allocator_name: "cpu" }

Compiled
I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 32
I tensorflow/core/common_runtime/direct_session.cc:83] Direct session inter op parallelism threads: 32
        0                Unknown (from Proto)(1-cpu)   569656800   569656800 cpu
        1                Unknown (from Proto)(1-cpu)  -569656800           0 cpu
        2               W_1/Initializer/Const(2-cpu)   569656800   569656800 cpu
        3                Unknown (from Proto)(3-cpu)        3600   569660400 cpu
        4                Unknown (from Proto)(3-cpu)       -3600   569656800 cpu
        5               b_1/Initializer/Const(4-cpu)        3600   569660400 cpu
        6                Unknown (from Proto)(5-cpu)     3600000   573260400 cpu
        7                Unknown (from Proto)(5-cpu)    -3600000   569660400 cpu
        8               W_2/Initializer/Const(6-cpu)     3600000   573260400 cpu
        9                Unknown (from Proto)(7-cpu)        4000   573264400 cpu
       10                Unknown (from Proto)(7-cpu)       -4000   573260400 cpu
       11               b_2/Initializer/Const(8-cpu)        4000   573264400 cpu
       12                Unknown (from Proto)(9-cpu)        4000   573268400 cpu
       13                Unknown (from Proto)(9-cpu)       -4000   573264400 cpu
       14              W_3/Initializer/Const(10-cpu)        4000   573268400 cpu
       25               Unknown (from Proto)(13-cpu)   569656800<PHONE_NUMBER> cpu
       26               Unknown (from Proto)(13-cpu)  -569656800   573268400 cpu
       28               Unknown (from Proto)(14-cpu)        3600   573272000 cpu
       29               Unknown (from Proto)(14-cpu)       -3600   573268400 cpu
       31                         b_1/Assign(16-cpu)        3600   573272000 cpu
       32                         W_1/Assign(17-cpu)   569656800<PHONE_NUMBER> cpu
       34               Unknown (from Proto)(15-cpu)     3600000<PHONE_NUMBER> cpu
       35               Unknown (from Proto)(15-cpu)    -3600000<PHONE_NUMBER> cpu
       37               Unknown (from Proto)(18-cpu)        4000<PHONE_NUMBER> cpu
       38               Unknown (from Proto)(18-cpu)       -4000<PHONE_NUMBER> cpu
       40               Unknown (from Proto)(19-cpu)        4000<PHONE_NUMBER> cpu
       41               Unknown (from Proto)(19-cpu)       -4000<PHONE_NUMBER> cpu
       48                         W_2/Assign(22-cpu)     3600000<PHONE_NUMBER> cpu
       49                         b_2/Assign(23-cpu)        4000<PHONE_NUMBER> cpu
       51                         W_3/Assign(24-cpu)        4000<PHONE_NUMBER> cpu
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 2 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 4 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 6 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 8 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 10 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 12 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 21 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 24 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 22 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 17 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 23 allocator_name: "cpu" }
I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorDeallocation { allocation_id: 16 allocator_name: "cpu" }

This is the script I used for testing:
import memory_util
memory_util.vlog(1)
import numpy as np
import tensorflow as tf

def get_layer(input_size, output_size, name):
    # supposedly loaded from a saved file
    W_val = np.random.normal(scale=0.1, size=(input_size, output_size)).astype(np.float32)
    W = tf.get_variable(name='W_{}'.format(name), shape=(input_size, output_size),
                        initializer=tf.constant_initializer(value=W_val, dtype=tf.float32),
                        dtype=tf.float32)
    b = tf.get_variable(name='b_{}'.format(name), shape=(output_size,),
                        initializer=tf.constant_initializer(value=0.1, dtype=tf.float32),
                        dtype=tf.float32)
    return W, b

g = tf.Graph()
with g.as_default():
    W1, b1 = get_layer(158238, 900, '1')
    W2, b2 = get_layer(900, 1000, '2')
    W3, b3 = get_layer(1000, 1, '3')

    init = tf.global_variables_initializer()
session = tf.Session(graph=g)
with memory_util.capture_stderr() as stderr:
    session.run(init)
memory_util.print_memory_timeline(stderr, ignore_less_than_bytes=1000)

Are there any updates on this issue? Has anyone been able to reproduce it?

memory_util only captures memory allocated by TF runtime, so this suggests the problem is elsewhere. Earlier you said that the extra memory comes from memory allocated by MergePartialFromCodedStream which could suggests that extra memory was allocated by protobuf library.
Note that you are initializing your layers with huge constants. Those constants get serialized into GraphDef protobufs. There might be a bug somewhere which causes memory allocated by this serialization not to be freed.
As a work-around, you could try not using constant nodes to initialize your variables. You can initialize them directly from numpy arrays by using tf.placeholder and providing value through feed_dict, for examples see https://github.com/tensorflow/tensorflow/issues/9821

@yaroslavvb wow, the difference is huge. Initializing the variables now takes less than 3GB of memory, that's 2x less than what the binary version was taking and 4x less than the compiled one (using tf.constant_initializer). That's a huge difference!
However, the placeholder initialization workaround is unpractical in my case as we have wrapper classes that do the loading for us which we just initialize using global_variables_initializer so it'll require some design changes which isn't very feasible right now.
I understand this is an issue with protobuf but how come when loading the variables in the compiled version of tensorflow we get so much more memory usage? Even though it's the same protobuf version used in both cases? Also should I file an issue with the protobuf team regarding this or are there ways to tune it?
In case someone needs it, here's the code I used:
import numpy as np
import tensorflow as tf

def get_layer(input_size, output_size, name):
    # supposedly loaded from a saved file
    W_pl = tf.placeholder(tf.float32, (input_size, output_size))
    W_val = np.random.normal(scale=0.1, size=(input_size, output_size)).astype(np.float32)
    W = tf.get_variable(name='W_{}'.format(name),
                        initializer=W_pl,
                        dtype=tf.float32)
    b = tf.get_variable(name='b_{}'.format(name), shape=(output_size,),
                        initializer=tf.constant_initializer(value=0.1, dtype=tf.float32),
                        dtype=tf.float32)
    return (W_pl, W_val), W, b

sessions = []
for i in range(3):
    g = tf.Graph()
    with g.as_default():
        W1_pl, W1, b1 = get_layer(158238, 900, '1')
        W2_pl, W2, b2 = get_layer(900, 1000, '2')
        W3_pl, W3, b3 = get_layer(1000, 1, '3')
        init = tf.global_variables_initializer()
    session = tf.Session(graph=g)
    session.run(init, feed_dict=dict([W1_pl, W2_pl, W3_pl]))
    print 'Loaded {}'.format(i)
    sessions.append(session)

BTW, putting huge constants into Graph has performance implications besides higher memory, accessing data is slower too. Also there's a 2GB limit which includes graph inlined constants, once you hit it, you won't be able to add any more operations to the graph.
I get around design issues by keeping global init_dict with links to all numpy arrays and passing it to global_variables_initializer.
If you really wanted to find out why there's a difference you could try to narrow problem further.
For instance, does it happen with Python protobufs as well or just with cpp?
To find out which protobuf you are running:
python -c "from google.protobuf.internal import api_implementation; print(api_implementation._default_implementation_type)"
Does it happen when the constant in question are getting read or is just the serialization step leaking memory? You can create large constant + small constant, and do "sess.run(small_constant)"

Thanks for the suggestion @yaroslavvb, sounds fair enough (and worth it given the significant memory/performance improvement).
Not sure it's worth it to look into the difference between the environments given that I found the main bottleneck now which is in the protobuf serilaization. For what it's worth the protobuf implementation type in my case is cpp.
As for your last question, I already tested with tf.random_normal_initializer instead of tf.constant_initializer with a numpy array and there's no issue at all.

It sounds as if the right thing to do is follow up with protobuf to investigate further, so I am closing this for now but please reopen if there are more TF issues.


Element identification

We should be able to switch the application to something like a "design" mode. This mode should be enabled by server via startup. If not enbled -> no chance to use this feature.
The client should show an icon or a button to switch to design mode (best place, top bar/menu). Button should en/disable design mode.
Feature: click or maybe only long-press -  on an element and show it selected (round rectangle, transparent color). Send a request to the server with name.
Use-case: Change the screen on-demand.

select a table -> show/hide columns
select an editor -> move editor down/up
select an editor -> add a new column to the table

We need to know "where we are" to change the screen.

Only sending the request and what kind of request is open, rest is implemented

You should support a simple API for this feature, because it's included in this project, but we will use it in a different project.

You can currently listen to the server sending yes/no if it is allowed to activate the design mode.
You can listen to the switch of on/off of the design mode itself.
You can also listen to the elements that are selected and when they are selected. So theoretically you can add listeners to all these 3 events.
Any more needed?

We need a feature to hide selection and design mode programmatically.

Hide selection is just disabling the active design mode or selecting null.
The ability to hide the design mode button is also possible.
I will implement a way to deactivate it permanently so that the server can't "allow" it via startup. Currently hiding is possible but will get overruled on the next startup again.

Overruling of application parameters now possible via AppManager, therefore design mode can be hidden/shown on demand and the server setting can be overruled before it is activated.


Provide a means to set request timeouts

It would be nice to have a means to set timeouts for the request calls going out, so in odd/unfortunate network conditions we don't end up blocking indefinitely.

I like this idea. Would you mind creating a PR to implement this? It should be pretty easy to do.

Closed by #12


Getting the aks quick start page, translation to es in the docs/pages…

…/es/installation folder
Signed-off-by: JOSE ALONSO LOPEZ ROMO<EMAIL_ADDRESS>Description
This PR fixes #
Notes for Reviewers
@jesuslerma tagged for review. Thanks man...
Signed commits

[x] Yes, I signed my commits.

The translation LGTM. Anyway @leecalcote  it looks that there is another changed file, that change is not critical. But please check it to give @alonsolopezr their second PR merged :) This is the other changed file in the PR:
docs/_data/toc.yml

@alonsolopezr when you get an extra file or two in your PRs, that typically means that your fork is out of sync and that you'll need to update it with the remote origin (upstream).

I have noticed this just yesterday.... but I do this fetch from upstream on master always, whenever I'm starting a new translate, and then the merge then the new branch (Jesus Lerma instructed that very well)... but I'm 'git add .' So I think this is my main mistake... I will just be adding only the files changed by me.... greetings and Have a great weekend

@alonsolopezr you're doing great. This is a very common issue that many people (including me) run into. 😄  Here is a set of instructions that will help - https://github.com/layer5io/meshery/blob/master/CONTRIBUTING-gitflow.md


Cumulative sensors not showing any data

Version of the custom_component
v1.4.0b7
Configuration
No logs.

Describe the bug
I've just switched over from the StealthChesnut integration to this. The following sensors are showing as unknown and have no data:

Load Power Total (sensor.load_power_total)
PV1 Power Total (sensor.pv1_energy_total)
PV2 Power Total (sensor.pv2_energy_total)

I appreciate I'm on a beta version so can't expect everything to be perfect. Thank you!
Debug log

No logs.

Is there nothing at all in your logs?
Do you have a H1 or an AC1?

Ah, do you have a friendly name set which has a space in it? If so, this is #106.

Which model of inverter are you using please?

My inverter is a FoxESS H1-5.0-E. There are no errors in the logs relating to the FoxESS - Modbus integration.
I've enabled debug logging and I just see logs of entries for each register being read, but nothing untoward.
The friendly name does indeed have spaces in it.

Yep, #106. That's due to be fixed before the next proper release.

Thanks! I'll change the friendly name and hopefully that fixes it.

Yep, it just needs to not have a space in it. HA silently fixes things for us most of the time, but not for the source entity of an integration sensor.

I'd expect all sensors to blow up?

I think HA must escape it for us most of the time.

The 3 sensors mentioned in the OP are working now I've omitted the friendly name. Thanks both.

Duplicate of #106


python36Packages.pytest-timeout: 1.3.2 -> 1.3.3

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/python3.6-pytest-timeout/versions.
meta.description for python36Packages.pytest-timeout is: '"py.test plugin to abort hanging tests"'.

Checks done (click to expand)

built on NixOS

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 1.3.3 with grep in /nix/store/m3ay4c9bymkps51i9qd3z4ljcj57pafk-python3.6-pytest-timeout-1.3.3

directory tree listing: https://gist.github.com/1f16d50bb216fd0714729903c24b7fef

du listing: https://gist.github.com/c654f2e6c6797caab10ffab256027e0e

Rebuild report (click to expand)

35 total rebuild path(s)
9 package rebuild(s)
9 x86_64-linux rebuild(s)
9 i686-linux rebuild(s)
8 x86_64-darwin rebuild(s)
9 aarch64-linux rebuild(s)
First fifty rebuilds by attrpath
devpi-client
devpi-server
ip2unix
python27Packages.can
python27Packages.canopen
python27Packages.pytest-timeout
python36Packages.can
python36Packages.canopen
python36Packages.pytest-timeout

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/m3ay4c9bymkps51i9qd3z4ljcj57pafk-python3.6-pytest-timeout-1.3.3 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
Or, build yourself:
nix-build -A python36Packages.pytest-timeout https://github.com/r-ryantm/nixpkgs/archive/a58f0014f53ae548bc6224abf977a14ef3fa83fa.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/m3ay4c9bymkps51i9qd3z4ljcj57pafk-python3.6-pytest-timeout-1.3.3
ls -la /nix/store/m3ay4c9bymkps51i9qd3z4ljcj57pafk-python3.6-pytest-timeout-1.3.3/bin

cc @makefu @costrouc for testing.

Success on x86_64-darwin (full log)
Attempted: python36Packages.pytest-timeout
Partial log (click to expand)
platform darwin -- Python 3.6.6, pytest-3.7.4, py-1.5.4, pluggy-0.7.1
rootdir: /private/tmp/nix-build-python3.6-pytest-timeout-1.3.3.drv-0/pytest-timeout-1.3.3, inifile: tox.ini
plugins: timeout-1.3.3
collected 35 items

test_pytest_timeout.py ...................................               [100%]

========================== 35 passed in 56.14 seconds ==========================
pytestcachePhase
/nix/store/zm3xrqgv9784n8qaff4f38135sgxdicg-python3.6-pytest-timeout-1.3.3

Success on aarch64-linux (full log)
Attempted: python36Packages.pytest-timeout
Partial log (click to expand)
platform linux -- Python 3.6.6, pytest-3.7.4, py-1.5.4, pluggy-0.7.1
rootdir: /build/pytest-timeout-1.3.3, inifile: tox.ini
plugins: timeout-1.3.3
collected 35 items

test_pytest_timeout.py ...................................               [100%]

========================== 35 passed in 72.93 seconds ==========================
pytestcachePhase
/nix/store/g9kz0b7jzy6pwcnbzb184dkmc94d4ixy-python3.6-pytest-timeout-1.3.3

Success on x86_64-linux (full log)
Attempted: python36Packages.pytest-timeout
Partial log (click to expand)
platform linux -- Python 3.6.6, pytest-3.7.4, py-1.5.4, pluggy-0.7.1
rootdir: /build/pytest-timeout-1.3.3, inifile: tox.ini
plugins: timeout-1.3.3
collected 35 items

test_pytest_timeout.py ...................................               [100%]

========================== 35 passed in 99.60 seconds ==========================
pytestcachePhase
/nix/store/m3ay4c9bymkps51i9qd3z4ljcj57pafk-python3.6-pytest-timeout-1.3.3

thanks @r-ryantm ,lgtm!


Make sure that the repo automatically pulls dependencies

it seems to me that this should probably include an embedded python interpreter and should statically include the npm/python libraries...

putting this off til we have something to build


[feature-request] Support for Apple Health Records

Describe the solution you'd like
Use Apple's FHIR models to fetch and store data from health records onto:

Firestore,
Google's Smart on FHIR.

https://github.com/apple/FHIRModels
NOTE: interest on features is gaged using reactions on this first post: e.g.  👍

Documentation on accessing Apple's Health Records:
https://developer.apple.com/documentation/healthkit/samples/accessing_health_records
Note that the iOS simulator has sample health record information that you can spoof and use.

Completed in #44


[nightly] generate types against bigcommerce/api-specs@067d52f

Types generated via scheduled GitHub Actions job against bigcommerce/api-specs@067d52f

Included in #48


Create full dump events

Once alma has created the files, it will fire a webhook. Use the AWS-lambda webhook listener developed as part of #645 to put an event on the AWS message queue, and subscribe to that queue from marc_liberation.
The local worker should then copy the files from the sftp server and create the dump and event objects.
amazon console is at princeton.edu/aws

Proposal:

Move work from https://github.com/pulibrary/alma_experimentation/pull/8/files and  https://github.com/pulibrary/alma_experimentation/pull/33/files into marc_liberation.
Update the webhook handler. If the webhook action indicates a full dump job is complete, enqueue the event body to an AWS SQS queue.
Use the sidekiq cron gem https://github.com/ondrejbartas/sidekiq-cron to schedule a job to run every 15 minutes as follows: ```
Check the queue and if there's an event invoke the service that will create the Dump and Event objects and copy the files.

This process does not need to do any indexing; full indexing is initiated manually.

SQS is free for the first million requests per month, so no issue there. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html#sqs-pricing

Why use a cron job rather than an event listening daemon similar to Figgy's CloudFixity::Worker?
https://github.com/pulibrary/figgy/blob/40ba9ea10f26693ec887c43a4b8b1905c4b1a5bc/app/services/cloud_fixity.rb

Thanks for mentioning this. It looks like that start from a rake task -- it would die if the box was rebooted? I wanted something that would be resilient to down time.

Docs for the QueuePoller

There's a systemd service file for it like we do for sidekiq workers, it spins back up if it dies.

https://github.com/pulibrary/princeton_ansible/blob/main/roles/pulibrary.figgy_pubsub_worker/templates/pubsub.service
Okay great this way seems better then since it can be evented in a true pub / sub way.

Here's what the webhook gives for the full publishing job:

@timestamp
@message

2020-10-21 20:18:29.376
2020-10-21T20:18:29.375Z ce8e48f5-5a0b-45b2-bd3f-ee6a0524390a INFO [dd.trace_id=2849026166357774790 dd.span_id=8780004452817738765] Message received: {"id":"1429676950006421","action":"JOB_END","institution":{"value":"01PRI_INST","desc":"Princeton University"},"time":"2020-10-21T20:18:21.913Z","event":null,"job_instance":{"id":"1429676950006421","external_id":"1429676980006421","name":"Publishing Platform Job FTP Dump","submitted_by":{"value":"AlmaAdmin","desc":"Administrator, Alma"},"submit_time":"2020-10-21T19:50:43.006Z","start_time":"2020-10-21T19:50:43.022Z","end_time":"2020-10-21T20:18:21.913Z","progress":100,"status":{"value":"COMPLETED_SUCCESS","desc":"Completed Successfully"},"status_date":"2020-10-21Z","alert":[{"value":"alert_general_success","desc":"The job completed successfully. For more information view the report details."}],"counter":[{"type":{"value":"label.new.records","desc":"New Records"},"value":"109435"},{"type":{"value":"label.updated.records","desc":"Updated Records"},"value":"0"},{"type":{"value":"label.deleted.records","desc":"Deleted Records"},"value":"0"},{"type":{"value":"c.jobs.publishing.failed.publishing","desc":"Unpublished failed records"},"value":"0"},{"type":{"value":"c.jobs.publishing.skipped","desc":"Skipped records (update date changed but no data change)"},"value":"0"},{"type":{"value":"c.jobs.publishing.filtered_out","desc":"Filtered records (not published due to filter)"},"value":"0"},{"type":{"value":"c.jobs.publishing.totalRecordsWrittenToFile","desc":"Total records written to file"},"value":"0"}],"job_info":{"id":"S1425772390006421","name":"Publishing Platform Job FTP Dump","description":"Publishing Platform Job","type":{"value":"SCHEDULED","desc":"Scheduled"},"category":{"value":"PUBLISHING","desc":"Publishing"},"link":"/almaws/v1/conf/jobs/S1425772390006421"},"link":"/almaws/v1/conf/jobs/M0/instances/1429676950006421"},"bib":null,"holding":null,"item":null,"portfolio":null,"representation":null}

Message expanded:
{
    "id": "1429676950006421",
    "action": "JOB_END",
    "institution": {
        "value": "01PRI_INST",
        "desc": "Princeton University"
    },
    "time": "2020-10-21T20:18:21.913Z",
    "event": null,
    "job_instance": {
        "id": "1429676950006421",
        "external_id": "1429676980006421",
        "name": "Publishing Platform Job FTP Dump",
        "submitted_by": {
            "value": "AlmaAdmin",
            "desc": "Administrator, Alma"
        },
        "submit_time": "2020-10-21T19:50:43.006Z",
        "start_time": "2020-10-21T19:50:43.022Z",
        "end_time": "2020-10-21T20:18:21.913Z",
        "progress": 100,
        "status": {
            "value": "COMPLETED_SUCCESS",
            "desc": "Completed Successfully"
        },
        "status_date": "2020-10-21Z",
        "alert": [
            {
                "value": "alert_general_success",
                "desc": "The job completed successfully. For more information view the report details."
            }
        ],
        "counter": [
            {
                "type": {
                    "value": "label.new.records",
                    "desc": "New Records"
                },
                "value": "109435"
            },
            {
                "type": {
                    "value": "label.updated.records",
                    "desc": "Updated Records"
                },
                "value": "0"
            },
            {
                "type": {
                    "value": "label.deleted.records",
                    "desc": "Deleted Records"
                },
                "value": "0"
            },
            {
                "type": {
                    "value": "c.jobs.publishing.failed.publishing",
                    "desc": "Unpublished failed records"
                },
                "value": "0"
            },
            {
                "type": {
                    "value": "c.jobs.publishing.skipped",
                    "desc": "Skipped records (update date changed but no data change)"
                },
                "value": "0"
            },
            {
                "type": {
                    "value": "c.jobs.publishing.filtered_out",
                    "desc": "Filtered records (not published due to filter)"
                },
                "value": "0"
            },
            {
                "type": {
                    "value": "c.jobs.publishing.totalRecordsWrittenToFile",
                    "desc": "Total records written to file"
                },
                "value": "0"
            }
        ],
        "job_info": {
            "id": "S1425772390006421",
            "name": "Publishing Platform Job FTP Dump",
            "description": "Publishing Platform Job",
            "type": {
                "value": "SCHEDULED",
                "desc": "Scheduled"
            },
            "category": {
                "value": "PUBLISHING",
                "desc": "Publishing"
            },
            "link": "/almaws/v1/conf/jobs/S1425772390006421"
        },
        "link": "/almaws/v1/conf/jobs/M0/instances/1429676950006421"
    },
    "bib": null,
    "holding": null,
    "item": null,
    "portfolio": null,
    "representation": null
}

The job has a "profile id" of<PHONE_NUMBER>006421 but that doesn't appear to be included in the webhook message. The only way to identify these jobs is by the profile name ("FTP Dump") prepended by the job type ("Publishing Platform Job").

Notes:

figgy-deploy is the profile we built for deploying the image server for figgy (the iiif-lambda stuff).  Do we want to keep this profile or use a different one? Maybe ask @kayiwa

There's a secret which is created and stored in the AWS secrets manage. The lambda has permission to get secrets, configured in the template.yml

Implementation is done but we want to create a diagram to document the architecture. Will close when that's finished.

Implementation is done but we want to create a diagram to document the architecture. Will close when that's finished.

Sequence diagram created at https://lucid.app/publicSegments/view/c63d2bcd-c5ba-4d92-aacc-56710638c065/image.pdf

Sequence diagram created at https://lucid.app/publicSegments/view/c63d2bcd-c5ba-4d92-aacc-56710638c065/image.pdf


.NET 6 Support

Asked for help to the ASP.NET Core team as there's a breaking change to our library with the introduction of https://github.com/aspnet/Announcements/issues/459
Help Link:
https://github.com/dotnet/aspnetcore/issues/31578#issuecomment-841066601

Found an alternative way to fix on our own
https://github.com/dotnet/aspnetcore/issues/31578#issuecomment-841834166

@soundaranbu any plan on the .net 6 support? the .net6 RTM will be on Nov 2021

Yes! Right now v1.5.0 works with  .NET 6 preview versions. But planning for v1.6.0 that has .NET 6 as base version.

New preview version v1.7.0-preview.1 released with .NET 6 support https://github.com/soundaranbu/RazorTemplating/releases/tag/v1.7.0-preview.1
Kindly try it & let me know if there are any issues.

Support for .NET 6 is now released as stable version 1.7.0.
https://github.com/soundaranbu/RazorTemplating/releases/tag/v1.7.0
Users of .NET 5 upgrading to .NET 6 should be using v1.7.0
Thanks!


Add a way to refresh table metadata

Currently table metadata is evaluated once lazily, but does not get refreshed. This means if you want to be guaranteed the latest values you may need to create a new Table instance.
A hacky workaround is to use:
Table._info = None

which will force a refresh next time the metadata is accessed.
We should just expose the above as a method.

Added in 640323a


[DataLoader] Deep copy ReadingService during DL2 initialization

Stack from ghstack:

#728
#747
-> #746

By creating a deep copy of the input ReadingService during the initialization of DataLoader2, we can allow the instance of ReadingService to store state, without worrying about the users passing the same ReadingService to initialize additional DataLoaders.
For instance, PrototypeMultiProcessingReadingService stores processes and datapipes as instance variables. In a single object of that ReadingService is used to initialize multiple DataLoaders, it could've led to conflicts and incorrect states. This should no longer be an issue with this change.
Relevant test are added in subsequent PR related to lengths and eventually PrototypeMultiprocessingReadingService
Differential Revision: D38868731

@nivekt has imported this pull request.  If you are a Facebook employee, you can view this diff on Phabricator.

We should put this information into the ReadingService interface file too. To make sure that RS developers know about potential deep copy.

One more though, we deepcopy RS, but we pickle-unpickle DP. I think we need to pick consistent approach.

One more though, we deepcopy RS, but we pickle-unpickle DP. I think we need to pick consistent approach.

Totally agree. And, I support deepcopy because it's more readable and even working with unpickable objects.

One more though, we deepcopy RS, but we pickle-unpickle DP. I think we need to pick consistent approach.

Can you elaborate on this? What make a consistent approach matter here?

Also, it seems like some internal ReadingService cannot be deep-copied.

Also, it seems like some internal ReadingService cannot be deep-copied. One alternative is for users to pass in a constructor instead of the object itself, but that will change the interface quite a bit.

It seems the Error comes from construction of process group in __init__ function. You might be able to move them into initialize function to make it working.

@nivekt has imported this pull request.  If you are a Facebook employee, you can view this diff on Phabricator.

@nivekt has imported this pull request.  If you are a Facebook employee, you can view this diff on Phabricator.

@nivekt has imported this pull request.  If you are a Facebook employee, you can view this diff on Phabricator.

@nivekt has imported this pull request.  If you are a Meta employee, you can view this diff on Phabricator.


HP.HPSupportAssistant version <IP_ADDRESS>41

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.1 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

So we are waiting for HP on this one then?

So we are waiting for HP on this one then?

WinGet will support interactive-only installers in the future which means installers that doesn't have silent install or the silent install works but some prompts aren't skippable will be supported.

@msftbot Interactive-Only-Installer

@msftbot Blocking-Issue

Request


Add interruption level option

Hi there. Great blueprint.
I have a feature request.
iOS and macOS devices have different levels of interruption for notifications. So it's possible to get the notification through despite I have everything disabled. With the strong implementation of focus modes in iOS and macOS this is a feature that is quite nice.
Most notifications are not extremely important, so the active level (default) is absolutely fine. This respects the users settings for the whole app. But there are three different modes according to the official documentation (https://companion.home-assistant.io/docs/notifications/notifications-basic/#interruption-level)

passive – This one is completely silent and just is there when you unlock the phone (great for battery low messages or something that can definitely wait)
active – The default as stated above (great for most notfications)
time-sensitive – This one breaks the focus even if Home Assistant is muted in this focus. (Great for close doorbells and stuff like that)
critical – This one breaks through everything, even mute. This is the highest level. (Great for leak sensors, security stuff or smoke alarm)

It would be nice if I could add this as a simple parameter for each notification I want to send.

Hi, thanks for your suggestion and nice to see people are using this blueprint!
This is not too hard to implement, but did you already try passing this parameter to your notifications using the data parameter? If so, why is that solution not sufficient?
Example:
service: script.notify_everybody_home
data:
  message: Test message
  title: Example title
  data:
    push:
      interruption-level: passive

My bad. I just didn't realize that this options exists. I still think it would be nice to have a dropdown for this, but I think it's not as urgent as I thought 😅


”Lunga” is listed in the drop-down menu

Description:
"Lunga" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Lunga" in the search bar.

Expected results
The location is listed in the drop-down menu as "Sat Lunga, COVASNA (TÂRGU SECUIESC)".


[Warhammer 4e Character Sheet] Fixes

Changes / Comments

Fixed issue where Characteristics Stat modifier was adding twice, this line is now also renamed "Misc" rather then "Modifier" and no long sets the roll modifier for characteristics rolls. Misc now correctly adds to Char bonuses.
Fixed Arcane Spell Target/Range/Duration errors when spellcasting

Roll20 Requests
Comments are very helpful for reviewing the code changes. Please answer the relevant questions below in your comment.

[ ] Does the pull request title have the sheet name(s)? Include each sheet name.
[x] Is this a bug fix?
[ ] Does this add functional enhancements (new features or extending existing features) ?
[ ] Does this add or change functional aesthetics (such as layout or color scheme) ?
[ ] If changing or removing attributes, what steps have you taken, if any, to preserve player data ?
[ ] If this is a new sheet, did you follow Building Character Sheets standards ?

If you do not know English. Please leave a comment in your native language.

Character Sheet Info Roll20 Internal Use only.


Refactor: Replace CJS code with ESM for code-generator index.ts

This is to partially address #12 issue with a small scope for now. I also added the node-notifier package as a dev dependency because while it's listed as an option peer dependency when I ran the tests it caused an error indicating it was required. I hope this is ok but please let me know if there is a more appropriate approach to resolve that.

I'm supr excited as this is the first PR here while it's actually under the radar!
Happy to have you here @clarkio 🌻
Will go through this obviously in a day or two

Just tested it and it works great! thanks @clarkio

@all-contributors please add @clarkio  for code

@clarkio You're the 1st on the contributor's board 💪
@rluvaton Welcome!


Can't ping <IP_ADDRESS> when running in qemu

Hello,
i tried to run this build in qemu as descriped in the readme qemu section. I'm getting this error. Maybe someone can help? Thanks in advance.

Hello_ from Rust!
More complex print ["foo", "bar"]
Result: 0, 1
Rust main thread: Thread { id: ThreadId(1), name: Some("main"), .. }
Main TLS before change: 13
Main TLS after change: 42
This is thread number 0, Thread { id: ThreadId(2), name: None, .. }
Inner TLS: 0
This is thread number 2, Thread { id: ThreadId(4), name: None, .. }
Inner TLS: 2
This is thread number 3, Thread { id: ThreadId(5), name: None, .. }
Inner TLS: 3
This is thread number 4, Thread { id: ThreadId(6), name: None, .. }
Inner TLS: 4
About to join the threads. If ESP-IDF was patched successfully, joining will NOT crash
This is thread number 1, Thread { id: ThreadId(3), name: None, .. }
Inner TLS: 1
Main TLS after threads: 42
Joins were successful.
I (16115) rust_esp32_std_demo: Testing critical sections
I (16145) rust_esp32_std_demo: Waiting for critical section
I (21135) rust_esp32_std_demo: Critical section acquired
I (21165) system_api: Base MAC address is not set
I (21175) system_api: read default base MAC address from EFUSE
I (21205) esp_idf_svc::eth: Driver initialized
I (21215) esp_idf_svc::eth: Initialization complete
I (21265) system_api: Base MAC address is not set
I (21265) system_api: read default base MAC address from EFUSE
I (21285) esp_idf_svc::eth: Stopping
E (21285) esp_eth: esp_eth_stop(288): driver not started yet
I (21285) esp_idf_svc::eth: Stop requested
I (21295) esp_eth.netif.netif_glue: 00:00:00:00:00:03
I (21305) esp_eth.netif.netif_glue: ethernet attached to netif
I (21305) rust_esp32_std_demo: Eth created
I (21435) esp_idf_svc::eth: Start requested
I (21435) rust_esp32_std_demo: Starting eth...
I (21505) esp_idf_svc::eth: About to wait for duration 20s
I (21515) esp_idf_svc::eth: Waiting done - success
I (21535) esp_idf_svc::netif::status: About to wait for duration 20s
I (22525) esp_idf_svc::netif::status: Got IP event: DhcpIpAssigned(DhcpIpAssignment { netif_handle: 0x3ffc42c4, ip_settings: IpInfo { ip: <IP_ADDRESS>, subnet: Subnet { gateway: <IP_ADDRESS>, mask: Mask(24) }, dns: None, secondary_dns: None }, ip_changed: true })
I (22565) esp_idf_svc::netif::status: Waiting done - success
I (22575) rust_esp32_std_demo: Eth DHCP info: IpInfo { ip: <IP_ADDRESS>, subnet: Subnet { gateway: <IP_ADDRESS>, mask: Mask(24) }, dns: Some(<IP_ADDRESS>), secondary_dns: Some(<IP_ADDRESS>) }
I (22585) rust_esp32_std_demo: About to do some pings for <IP_ADDRESS>
I (22585) esp_idf_svc::ping: About to run a summary ping <IP_ADDRESS> with configuration Configuration { count: 5, interval: 1s, timeout: 1s, data_size: 56, tos: 0 }
I (22615) esp_idf_svc::ping: Ping session established, got handle 0x3ffc4854
I (22645) esp_idf_svc::ping: Ping session started
I (22655) esp_idf_svc::ping: Waiting for the ping session to complete
I (23665) esp_idf_svc::ping: Ping timeout callback invoked
I (23665) esp_idf_svc::ping: From ??? icmp_seq=1 timeout
I (24675) esp_idf_svc::ping: Ping timeout callback invoked
I (24675) esp_idf_svc::ping: From ??? icmp_seq=2 timeout
I (25695) esp_idf_svc::ping: Ping timeout callback invoked
I (25695) esp_idf_svc::ping: From ??? icmp_seq=3 timeout
I (26705) esp_idf_svc::ping: Ping timeout callback invoked
I (26705) esp_idf_svc::ping: From ??? icmp_seq=4 timeout
I (27715) esp_idf_svc::ping: Ping timeout callback invoked
I (27715) esp_idf_svc::ping: From ??? icmp_seq=5 timeout
I (27715) esp_idf_svc::ping: Ping end callback invoked
I (27725) esp_idf_svc::ping: 5 packets transmitted, 0 received, time 20049ms
I (27735) esp_idf_svc::ping: Ping session stopped
I (27735) esp_idf_svc::ping: Ping session 0x3ffc4854 removed
I (27745) esp_idf_svc::eth: Stopping
I (27755) esp_idf_svc::eth: Stop requested
I (27785) esp_idf_svc::eth: Stopping
E (27785) esp_eth: esp_eth_stop(288): driver not started yet
I (27785) esp_idf_svc::eth: Stop requested
I (27795) esp_idf_svc::eth: Driver deinitialized
I (27795) esp_idf_svc::eth: Dropped
I (27805) esp_idf_svc::eventloop: Dropped
Error: Pinging IP <IP_ADDRESS> resulted in timeouts

My lo looks like this:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet <IP_ADDRESS>/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

found out that qemu does not support ping. Sorry for the fuss.


Update download.md: add integration monitor

🏷 Type of documentation

[ ] new bid adapter
[ ] update bid adapter
[x] new feature
[ ] text edit only (wording, typos)
[ ] bugfix (code examples)
[x] new examples

📋 Checklist

[ ] Related pull requests in prebid.js or server are linked -> Paste link in this list or reference it on the PR itself
[ ] For new adapters check submitting your adapter docs

how does it look now?


chore: add google/shopping/type/types.proto when generating shopping APIs

In this PR:

add google/shopping/type/types.proto when generating from google/shopping.

Hold this PR until https://github.com/googleapis/sdk-platform-java/issues/2018 is resolved.


[Hub Generated] Review request for Microsoft.Maps to add version stable/2021-02-01

This is a PR generated at OpenAPI Hub. You can view your work branch via this link.
Changelog
Please ensure to add changelog with this PR by answering the following questions.

What's the purpose of the update?

[ ] new service onboarding
[x] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[ ] Other, please clarify

When you are targeting to deploy new service/feature to public regions? Please provide date, or month to public if date is not available yet.

April 2021

When you expect to publish swagger? Please provide date, or month to public if date is not available yet.

April 2021

If it's an update to existing version,  please select SDKs of specific language and CLIs that require refresh after swagger is published.

[ ] SDK of .NET (need service team to ensure code readiness)
[ ] SDK of Python
[ ] SDK of Java
[ ] SDK of Js
[ ] SDK of Go
[ ] PowerShell
[ ] CLI
[ ] Terraform
[ ] No, no need to refresh for updates in this PR

Contribution checklist:

[x] I commit to follow the Breaking Change Policy of "no breaking changes"
[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

[x] Ensure to check this box if one of the following scenarios meet updates in the PR, so that label “WaitForARMFeedback” will be added automatically to involve ARM API Review. Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs, all “removals” and “adding a new property” no more require ARM API review.

Adding new API(s)
Adding a new API version
[x] Ensure to copy the existing version into new directory structure for first commit (including refactoring) and then push new changes including version updates in separate commits. This is required to review the changes efficiently.
Adding a new service

[x] Please ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If there are following updates in the PR, ensure to request an approval from Breaking Change Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in stable version
[ ] Removing properties in stable version
[ ] Removing API version(s) in stable version
[x] Updating API in stable or public preview version with Breaking Change Validation errors
[x] Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.
Please follow the link to find more details on PR review process.

/azp run unifiedPipeline

Just making sure that this PR is removing endpoints, is it expected?

his PR is removing endpoints, is it expected?

Yes, removing privateAtlas is expected, we are retiring the resourceType and it being replaced with "creators" type.

@jorgecotillo is there some action which I need to take here? This PR is addressing a bunch of PLR criteria which is on deadline, I see dotnet sdk build is failing but I'm not sure if that's caused from my PR.

@ArcturusZhang - is there any action which is required from the dotnet sdk build failure here for me?

@varunkumta as per your comments made changes, but now we have cross version breaking changes. Any recommendations what I can do here?

/azp run

@ArcturusZhang
Business Justification:
MAPS Creator is a PLR item that was recently onboarded. The GA is 5/21, but will use a new swagger API version containing the new SKUs. Once that swagger is merged we can kick off the GA work for Azure CLI, PowerShell and Terraform.
Creator is a resource moving from Preview to GA. It has not been part of a SDK release previously, so no customers SDK customers should be broken as a result of the change. The change removes an incorrect pattern and moves the RP to the ARM recommended best practice in preparation for the GA release. Can you please look and sign off?

For python, the breaking change is ok

Hi @stack111 I understand, but unfortunately I cannot sign off on the breaking changes issue - it can only be signed off by the breaking change review board. Could you please follow the guidance above to open a breaking change review request with your justification?

@ArcturusZhang - we have the breaking changes approved. Please continue forward.

@ArcturusZhang - sorry I accidently clicked the wrong button and closed the PR, I reopened but it added the CI python tag back. Please approve as soon as we can.

We could merge this once we revert the changes on package.lock @stach111


Git as a storage backend

I'd love to see git support in ark similar to https://github.com/pieterlange/kube-backup.  Generally, kubernetes is great at the present state of the system, but I think ark could give us some very powerful cluster history inspection if paired with git.  Imagine that instead of creating a record in an object store, it'd commit changes to a git repo and push it to github.
Some use cases that could be handy:

see an audit log of cluster changes in the same format as other software (eg. in github) irrespective of what tools people are using to create resources in the cluster and answer questions like: how many pods were running last thursday and what version were they?
use git bisect to identify the change that created an issue in production
diff staging and production clusters to see if they have diverged
use git tools to pull out unwanted cluster changes and reapply the config

I would absolutely love if this could run constantly by watching the API and writing out changes as they happen, but even a regular snapshot could be very useful.
This has been on my mind for a while as a potential standalone project, but it seems like it might fit well within ark.  I can try to take a crack at it if there's interest.  I also recognize this might be a bit of scope creep. ;)
Looking forward to hear your thoughts!

Hi @bdimcheff, thanks for suggesting this! I think it's a very interesting idea. I'm not sure we're ready to try to do continuous watches, but I do think we could extract an interface to support custom backup persistence formats. Something along the lines of BackupStarted, PersistItem, BackupCompleted, RetrieveItem. Just some initial thoughts here, and more to come over the next few weeks once we decompress from KubeCon 😄.
cc @jbeda

The current backup process looks like this:

A Backup resource is created
The BackupController in the ark pod sees the new Backup
The BackupController creates a temp file for the backup tarball
The BackupController tells its backupper (a kubernetesBackupper instance) to Backup(backup *api.Backup, backupFile, logFile io.Writer, actions []ItemAction), passing in the backup tarball temp file (as an io.Writer)
The kubernetesBackupper does its thing, backing up everything into the backupFile io.Writer

Now for a few comments & questions:
If we do this, the backupFile in Backup() probably needs to become an ItemPersister interface (name TBD).
We're starting to think about the idea of a "backup target" as a CRD that represents where the backup tarball is stored. For example, perhaps you decide you want to store your backup in multiple s3 buckets (in different regions), or in both s3 and azure. I want to make sure we keep your idea in mind as we flesh out the backup target design.
Should setting the storage backend be per backup, or should it apply to all of ark (i.e. via the ark config)? How are backups differentiated in git storage?
Ark currently places the temp files for the backup tarball and the backup log file in the default temp dir (via ioutil.TempFile("", "")). This means /tmp inside the ark container, which currently is part of the container's file system. Regardless of tarball vs git, should we have a specific root dir we should use to place temp files, git repos, etc?
For git, does the target repo need to exist first - is the first ark operation a git clone?
When the backup is complete, I guess we do a git commit and a git push?

@bdimcheff just getting back to this. Talking about some design options in #ark-dr in the Kubernetes slack.

@ajbouh it's not currently on the roadmap. If we decide to proceed, we need to find a design that allows for both the current tarball approach and a git approach to work easily, without a lot of if/else blocks.

for the time being, I've settled on using kube-backup to snapshot the yamls hourly and then running ark less often to do a "full" backup... it's been working pretty well for me for the last 6 weeks or so.

@bdimcheff, have you thought about (or seen) any continuous options? That is, a system that will keep a live backup of yamls as they are updated?

Thanks for this request. We are not currently prioritizing this item within our roadmap, and as such will be closing out the issue. Please feel free to reach out as needed in this issue or a new one.


Getting Started Project Fails On Compilation: VS For Mac

I am following the getting started guide and downloaded the  SqueakyApp.   I opened it in Visual Studio for Mac and I am getting the following error:
/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS/Xamarin.iOS.Common.targets(3,3): Error MSB4044: The "EmbedMobileProvision" task was not given a value for the required parameter "ProvisioningProfile". (MSB4044) (SqueakyApp.iOS)
I do not have an Apple ID set up on this Mac

Note that it does compile when targeting Android and unloading the iOS project.

Yes, just the simulator deploy
Yes, I ran the second step right before reporting the issue
.fsproj file is attached - I had to rename so the Mac would allow me to upload it - just change the extension back to .fsproj

Thanks
SqueakyApp.iOS.docx

This does not seem to be related to the project. Your fsproj is identical to mine.
Tried on another clean Mac as well, no luck reproducing...
Could you share your version info from VS for Mac please?
Visual Studio [upper left] > About Visual Studio > Show Details > Copy Information
Also do you have another Mac where you can try to reproduce the issue?
Thanks

I deleted the fold and pulled it down again
and now a new problem:
/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS/Xamarin.iOS.Common.targets(89,89): Error MSB4057: The target "_DetectSigningIdentity" does not exist in the project. (MSB4057) (SqueakyApp.iOS)


Ignore boss jumps and make levels/second actually accurate

@Novex please bump version in this PR.


FEAT-#4320: Add connectorx as an alternative engine for read_sql

What do these changes do?

Add connectorx as an alternative engine for read_sql. #4320
I tested locally by setting the env var READ_SQL_ENGINE to connectorx and it works fine. However, when I set the config by ReadSqlEngine.put("connectorx") (like here), the result of ReadSqlEngine retrieved in the PandasSQLParser is pandas. It's very likely that I missed some setup of the config. Can you take a look?

[x] commit message follows format outlined here
[x] passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
[x] passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
[x] signed commit with git commit -s
[x] Resolves #4320
[x] tests added and passing
[ ] module layout described at docs/development/architecture.rst is up-to-date
[ ] added (Issue Number: PR title (PR Number)) and github username to release notes for next major release

@wangxiaoying We are dealing with some CI environment issues, working on fix in #4364

@wangxiaoying We are dealing with some CI environment issues, working on fix in #4364

Got it! Please let me know if you need any actions from my side.

@wangxiaoying I think it's all set, if you can rebase on the master branch, CI should pass!

Hi @YarShev , thanks for the suggestions. I tried to test on the capital letter solution but it still has the same issue. I have added two debug messages in the latest commit. As the screenshot below, the value found in SqlDispatcher is Connectorx while in PandasSQLParser is Pandas.

You can find my test script here.

@wangxiaoying, the issue is that parse method is run in worker processes, which do not see an updated ReadSqlEngine value. As a solution, you can try passing a parameter called read_engine, for example, in the parse method and check the parameter value on Connectorx. That should work as intended.

@YarShev , I have updated the code following your suggestions. And the issue is resolved after passing the engine configuration to the parser!

@YarShev , I have fixed the comments. Please let me know if you need other actions from my side!

@wangxiaoying, I've run CI and lint (commit) job failed. To pass it you should format the commit message as follows:
git commit --amend

in an opened up editor you should put the lines in the following order
FEAT-#4320: Add `connectorx` as an alternative engine for `read_sql`

Co-authored-by: Devin Petersohn<EMAIL_ADDRESS>Co-authored-by: Yaroslav Igoshev<EMAIL_ADDRESS>Signed-off-by: wangxiaoying<EMAIL_ADDRESS>
after doing that push the changes
git push -f

@YarShev , I've updated the commit message and applied those code suggestions. Can you take a look again?

Some failures have occurred in CI. I restarted tests to see if it helps.

@wangxiaoying it looks like a SQL syntax error occurred in CI: https://github.com/modin-project/modin/runs/5988352034?check_suite_focus=true

@devin-petersohn indeed, the reason is that we remove the clauses such as order by and sort by when constructing the count query since they usually do not affect the # rows. However, in this mssql case the order by is required with offset. We will have a bug fix on this.

@devin-petersohn @YarShev , we have released an alpha version for the bug fix and requirement files are updated. Can you please try a gain?

I ran tests. Will see how it goes.

@YarShev Thanks for the example. I have updated the release note. I added to "Performance enhancements" but let me know if you think other sections suit this issue better : )

I think it is okay. Please fix conflicts.

I think it is okay. Please fix conflicts.

@YarShev Done


maxCharge/Discharge Current are wrong

Hello!
I have two batteries with 100A maxChargeCurrent each and 100A maxDischargeCurrent each
in the BatteryAggregator the limits are set to 100A each
The batteries are connected in parallel. Shouldn't the limits ever be 200A?

Good catch, fixed.


CC0021 generates a use NameOf for a constant defined as itself

#Bug
CC0021 suggests using a constant definition name as itself which does not compile.
'the code that reproduces the bug
Class MyClass
    Public Const MyConst="MyConst"
End Class

Current output after fix applied:
'code fixed incorrectly
Class MyClass
    Public Const MyConst=NameOf(MyConst)
End Class

Error:
BC30500	Constant 'MyConst' cannot depend on its own value.

@CZEMacLeod This analyzer checks if value of variable it´s equal your name and then suggest a fix;

I'm not quite sure what you mean by that.
The problem is that CC0021 from the NuGet package version on NuGet.org in visual basic suggests replacing a text string value for a constant with the same name. If you accept this change, you get error BC30500 which means the code does not compile.

@CZEMacLeod It's make sense, A solution it's ignore rule in variable declaration, when name and value are same. What do you think about ?

Yeah - That is what is required. Do not offer CC0021 when the nameof operator would point to the item you are declaring.


Feature Request

This is a Nice package with very good performance Nice work Done here ...
As I see Right Now Only QR Code Is Supported:
class QRCodeReader implements Reader {
can we have support of other formats too ?
https://barcode.tec-it.com/en/
Thanks !

@xvrh

New format can easily be added if this port proves useful.

It's useful. What would it take to get other barcode support ported?

Excuse me, library can support read qr code from image for web platform?

Please, 1D barcode reader is also very useful!


Floating point precision for log10

Hi there,
while experimenting with Uiua, I found that e.g. ₙ10 1000 gives the incorrect result of 2.9999999999999996 instead of 3, caused by a rounding error:
ₙ10 1 = 0
ₙ10 10 = 1
ₙ10 100 = 2
ₙ10 1000 = 2.9999999999999996
ₙ10 10000 = 4
ₙ10 100000 = 5
ₙ10 1000000 = 5.999999999999999
ₙ10 10000000 = 7
ₙ10 100000000 = 8
ₙ10<PHONE_NUMBER> = 8.999999999999998

The following test case should uncover the problem:
∵(⍤"ₙ10 failed!" =ₙ10 ⁿ∶10 .) ⇡11

This error can be reproduced in the pad as well as in a locally installed interpreter.
It seems that there is an Issue with Rust's underlying f64/f32 log and that there exists an integer variant that circumvents/fixes the problem. However, https://github.com/uiua-lang/uiua/blob/f19ec8b7e6538ddbab5c0a73ec930c6f9c01b978/src/algorithm/pervade.rs#L884 seems to define the binary operation based on the floating point type f64. I suppose a possible solution would be a special implementation that explicitly uses ilog instead of log in the case that the supplied value is integral? Unfortunately I am a complete noob concerning Rust, so I can not provide you with a pull request. But I hope my analysis helps :grin:
Jegge

Hi Jegge!
tl;dr: Short of implementing a new log function, implementing symbolic algebra optimizations, or switching to a new floating/fixed point type, the root problem cannot be fixed. A practical solution might be to round displayed outputs to an agreeable decimal place, and have slightly more flexible floating point equality checking behind the scenes.
I've done some investigating, and it looks like the default number type is f64. This means that a drop-in replacement won't work because ilog only works on integer types. I did attempt your proposed solution though, I've copied my work down below.
As referenced in the rust version of this issue, the inaccuracy boils down to the reality that IEEE doesn't have a standard for this because there are many trade-offs when designing intrinsic math functions. Rust's implementation used b.ln()/a.ln(), which ends in an inherently inaccurate floating point division.
I think a practical solution to this would be rounding floats in the output to a reasonable decimal place (and adding a feature/flag to disable this behavior). Python took this approach, and standardized rounding displayed output as of v3.1 according to the docs. They also provide a math.isclose to avoid inequalities behind the scenes. Rust's own assert_float_eq module is equipped with many near-equality checkers.
A Partial Solution based on Your Suggestion
I expanded the macro you referenced and added ilog for the u8 byte type, but this is useless unless byte arithmetic is enabled for log which I don't yet know how to do. What's more, a single byte wouldn't even be able to do log10(1000).
My implementation of the log module is below:
pub mod log {
    use super::*;
    #[cfg(feature = "bytes")]
    pub fn byte_byte(a: u8, b: u8) -> f64 {
        b.ilog(a) as f64
    }
    #[cfg(feature = "bytes")]
    pub fn byte_num(a: u8, b: f64) -> f64 {
        let a = f64::from(a);
        b.log(a)
    }
    #[cfg(feature = "bytes")]
    pub fn num_byte(a: f64, b: u8) -> f64 {
        let b = f64::from(b);
        b.log(a)
    }
    pub fn num_num(a: f64, b: f64) -> f64 {
        b.log(a)
    }
    #[cfg(feature = "complex")]
    pub fn com_x(a: Complex, b: impl Into<Complex>) -> Complex {
        let b = b.into();
        b.log(a)
    }
    #[cfg(feature = "complex")]
    pub fn x_com(a: impl Into<Complex>, b: Complex) -> Complex {
        let a = a.into();
        b.log(a)
    }
    pub fn error<T: Display>(a: T, b: T, env: &Uiua) -> UiuaError {
        env.error(format!("Cannot get the log base {0} of {1}", b, a))
    }
}

Hi SybeBlue,
thank you for your reply. Is #[cfg(feature = "bytes")] a compile-time option? To be honest, I do not completely understand your proposed solution. But in the meantime I've also given this a thought and I whipped up two variants of my initial solution. I've tried to format them rust-like, but since I've never coded in this language, don't expect them to compile ^^ :
The first variant checks for an integral a and casts to i64 if necessary. For the sake of simplicity I've used == for the f64, I am aware that  maybe something as float_eq might be needed.
pub fn num_num(a: f64, b: f64) -> f64 {
  if (a.fract() > 0) {
    a.log(b)
  } else if (b == 2) {
   (a as i64).ilog2() as f64
  } else if (b == 10) {
   (a as i64).ilog10() as f64
  } else {
   (a as i64).ilog(b as i64) as f64
  }
}

The second variant should be slightly faster than the first. According to the docs f64::log2 and f64::log10 improve the precision for these special bases and should be used instead of log:

The result might not be correctly rounded owing to implementation details; self.log2() can produce more accurate results for base 2, and self.log10() can produce more accurate results for base 10.

This variant might be a reasonable middle ground:
pub fn num_num(a: f64, b: f64) -> f64 {
  if (b == 2) {
   a.log2()
  } else if (b == 10) {
   a.log10()
  } else {
   a.log(b)
  }
}

One thing I would be worried about with solutions that do checks like this is speed.
Adding branches to a pervasive operation will slow it down, since it has to check on every element.
How much slower it would get I'm not sure.

Well I think that the speed penalty should not be an issue, since it's a simple test, especially in variant two. If I were to choose, I would prefer correctness over speed.

I've noticed another problem with some of the implementations above.
ilog rounds, which isn't appropriate.

I ran into this issue today,
For what it's worth, I would think about looking at symbolic algebra optimization long term.
I really like Uiua for self contained calculations, and I honestly dream of having a graphing calculator that has the Uiua symbols built into the keypad.
I understand that this suggestion might be a gargantuan effort, but I'd hope the Rust community start building something like this into the core of the language.
Just an opinion.


Add Streamlit dashboard integration example

Task
Add an example of Streamlit dashboard integration. Allows users to navigate and visualize HTML reports over multiple projects.
Short description
This example runs a Streamlit app that provides a simple UI to navigate over Evidently reports. Users may lunch the app, create a new project directory, add more reports, and switch between projects, monitoring periods, and reports.

Details
The streamlit_dashboard app generates a Dashboard by parsing Evidently reports in the projects/ directory.
There is a convention for the project directory that makes the app extendable. It expects that each project contains reports/directories with  .html files inside.
projects/
	bike-sharing/
		reports/
        your-project/
		reports/
...

Inside the reports/ directory, you may organize reports in a different way

by dates
by model names

This tutorial example uses the monitoring period start and end dates. For example, a directory named 2011-02-12_2011-02-12 has reports generated for the period from 02 February 2011 to 12 February 2011.

As a result, you may store data and model monitoring reports in different directories and select them from the UI.

It's also possible to have an additional level of reports hierarchy by adding sub-directories in the reports/ directory. In this case, reports are displayed in Tabs. For example, reports inside data_quality, model_performance folders are displayed in tabs:

Notes

It's recommended to use Python >= 3.9.12
The Streamlit version 1.19.0 doesn't work with Python 3.9.7

Hi @mnrozhkov ,
Thank you! It looks amazing!
I have one request regarding the example structure. I see that there is .gitignore file included in almost each folder. Could you please kinda merge them all to create the ultimate .gitignore and place this file into streamlit_dashboard folder, so that we have only one .gitgnore file in the whole example?

Hi @mnrozhkov , Thank you! It looks amazing!
I have one request regarding the example structure. I see that there is .gitignore file included in almost each folder. Could you please kinda merge them all to create the ultimate .gitignore and place this file into streamlit_dashboard folder, so that we have only one .gitgnore file in the whole example?

Thanks, @emeli-dral! I removed 3 redundant .gitignore files. Others are required to keep the empty folders in the example folder. This helps to simplify user workflow.


Test connection IDEA 15

java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 path $

Fixed in v0.1.1


Different audio input/output

Can I use another audio system with the jack? Or does it have to be a usb audio recorder?
Kind regards!!

If it's reachable with python-sounddevice it should work.


[BUG] Characters' Encoding Problems

My system

odbc Package Version: 2.4.4
ODBC Driver: HFSQL v27 for Windows (both x32/x64 versions, supports Unicode)
Database Name: HFSQL Client/Server (HyperFileSQL by PCSOFT, connection string details: https://doc.windev.com/en-US/?3044179)
Database Version: v27
Database OS: I don't have this information, since it's on a distant machine (but probably Windows..)
Node.js Version: 16.17.0
NPM Version: 8.15.0
Node.js OS: Windows 10 Entreprise (Build 19044)

The bug
There is a problem with characters' encoding in the returned data. In fact, the text is in French and the letters with accents are not correctly displayed (é, è, ç, à, ù, ê, ô, etc..), and any of these non-ascii charcters results in this "�". I've searched a lot for the reasons and possible solutions of this problem, tried everything I could, but with no success... The ODBC connection's encoding seems to be ANSI (must be ASCII), however the driver should support Unicode and be able to use the wide charset API, as it is stated on the official page of HFSQL installation procedure: https://doc.windev.com/en-US/?3044179

What I tried to do

If I understood correctly, the encoding depends on the ODBC connections itself, and it can be either ANSI or Unicode. So, I tried to change the connection string, adding keywords such as CCSID=1208, Charset=UTF8 etc, to explicitly set the encoding to Unicode. But to my regret, these keywords are specific to the Database, and couldn't find any for HFSQL in particular...
I tried these with different types/combinations of the connection string:

 `DRIVER={${driver}};Server Name=${host};Server Port=${port};Database=${database};UID=${UID};PWD=${PWD}`

 `DSN=${DSN};UID=${UID};PWD=${PWD}` // rest of the info is specified in the Driver Manager

 `DSN=${DSN}` // rest of the info is specified in the Driver Manager

I also tried to debug with Windows' ODBC Driver Manager's Tracing function enabled. The log file seems to be ok, except 1 ODBC function: *
node  server    2768-5f80	ENTER SQLDriverConnectW
	HDBC                0x0000029B2E6CF631
	HWND                0x0000000000000000
	WCHAR *             0x00007FF4D9C562E0 [      -3] "******\ 0"
	SWORD                       -3
	WCHAR *             0x00007FF4D9C562E0
	SWORD                       -3
	SWORD *             0x0000000000000000
	UWORD                        0 <SQL_DRIVER_NOPROMPT>

node  server    2768-5f80	EXIT  SQLDriverConnectW  with return code 1 (SQL_SUCCESS_WITH_INFO)
	HDBC                0x0000029B2E6CF631
	HWND                0x0000000000000000
	WCHAR *             0x00007FF4D9C562E0 [      -3] "******\ 0"
	SWORD                       -3
	WCHAR *             0x00007FF4D9C562E0 <Invalid buffer length!> [-3]
	SWORD                       -3
	SWORD *             0x0000000000000000
	UWORD                        0 <SQL_DRIVER_NOPROMPT>

	DIAG [IM006] [Microsoft][Gestionnaire de pilotes ODBC] �chec SQLSetConnectAttr du pilote (0)

(*) The last message is in French, and says IM006[Microsoft][ODBC Driver Manager] Driver's SQLSetConnectAttr failed. And even here in the log, it displays � instead of characters !
I don't know though why the connection string (3rd parameter of the function) is "******\ 0" and why its length (4th parameter) is equal to -3.
The Invalid buffer length! error should correspond to the following SQLDriverConnect SQLSTATE:

SQLSTATE
Description
Explanation

HY090
String or buffer length that is not valid
The value specified for StringLength1 is less than 0, but not equal to SQL_NTS. The value specified for BufferLength is less than 0.

But I don't understand this error..

I tried to define 'UNICODE' in defines of the binding.gyp file, but that did not help. P.S. I defined 'DEBUG' as well, but I do not get any node-odbc package's debug messages in the terminal when running my app with npm start (i'm using Express.js).

Expected behavior
Correctly displayed characters in the query results.

pls help T_T

I don't know though why the connection string (3rd parameter of the function) is "******\ 0" and why its length (4th parameter) is equal to -3.

-3 is SQL_NTS, ie a null-terminated string. It's likely traced with asterisks to prevent tracing out a password.
@markdirish I wonder if we need some sort of connection option which forces any character strings to be bound as SQL_C_WCHAR instead of SQL_C_CHAR, that way we can have a consistent encoding. Seems too many databases on Windows still using ANSI encoding for SQL_C_CHAR.

I see now that the invalid buffer length message is getting printed for the 5th parm, which is the output string. Indeed, SQL_NTS is an invalid value for an output buffer length. The application needs to pass in a buffer size and the driver should be returning the actual length of the data available to return (which may be bigger than the input buffer size), but neither should ever be SQL_NTS.
Of course, node-odbc always passes NULL/0 for the output buffer: https://github.com/markdirish/node-odbc/blob/55037241591beee343b3d9ce2e4f37ca8426255e/src/odbc.cpp#L416-L417
I'm guessing the discrepancy is being injected by the ODBC Driver Manager for some reason (though you'd think it would know better than passing in SQL_NTS...)

I'm guessing the discrepancy is being injected by the ODBC Driver Manager for some reason

Might this be related to the fact that in my ODBC.INI file there is only config for x32 driver ?
[ODBC 32 bit Data Sources]
DonneesSql=SQL Server (32 bit)
Winlassie=HFSQL (32 bit)
WinlassieSys=HFSQL (32 bit)
[DonneesSql]
Driver32=C:\WINDOWS\system32\SQLSRV32.dll
[Winlassie]
Driver32=C:\Program Files (x86)\Common Files\PC SOFT\27.0\ODBC\Win32x86\wd270hfo.dll
[WinlassieSys]
Driver32=C:\Program Files (x86)\Common Files\PC SOFT\27.0\ODBC\Win32x86\wd270hfo.dll

Even if I use the x64 version of the Windows ODBC Driver Manager, it creates only x32 entries in the .INI file for some reason....

May be it would be worth to mention:
When using DBeaver to connect to the DB via ODBC, all characters are displayed correctly. And I even can export data (as csv for example), but when choosing an encoding of the data, the UTF-8 option results in weird characters (using Excel to open the export file, example: Veuillez Ã©tablir instead of Veuillez établir)... However, the WINDOWS-1252 encoding gets the job done...

@kadler The problem was that the char encoding in the Database was set to WINDOWS-1252 for some reason 😅 , so I just fixed that by adding some conversions directly in the SELECT of my SQL queries:
CAST(CONVERT(column, 'WINDOWS1252', 'UTF8') AS VARCHAR(1024))

The first function CONVERT, as its name implies, converts the char encdoing and returns an SQL_LONGVARBINARY.
Then the 2nd one CAST represents the binary sequence as a string.

Thanks for the help !

@marbrex Glad you found a solution/workaround. I'm not sure all databases would have such conversion functions, however, so we'll probably still need to look in to a more universal solution.


Updated README, added npm install option.

Added github install from #153, cleaned up text using suggestions from #141.

@ljoy913, please add bin install limitations, as discussed.


adding to contributor page

not sure if i'm doing it correctly :( and how do i add a photo?

only the route is not enough, we are not accepting these anymore.. we might add everyone at the end


fix according to Sherine's comments

should be good for submission now.

Go for it!
--
Titus Brown<EMAIL_ADDRESS>
On Mar 9, 2015, at 2:01 PM, Qingpeng Zhang<EMAIL_ADDRESS>wrote:
should be good for submission now.
—
Reply to this email directly or view it on GitHub.


US101469: Wrap all tests in a describe.

Trying to reduce flakiness in sauce tests.

Still failed with the document.getElementById(...).create error.


Contain scope in unit tests

The unit tests were breaking in IE because the presence of var moment
was stamping over window.moment.
To fix this, I've wrapped the unit tests and lang tests in a closure.

I merged another pull request before this one, so it couldn't be automatically merged. I've updated it manually.