Possible memory leak when using Google Sheets API

There seems to be a memory leak when using the google-api-client with GSheets.
Environment:
$ python --version
Python 3.6.4
$ pip show google-api-python-client
Name: google-api-python-client
Version: 1.7.3

Here's a simple reproducer (without a .client_secret.json):
#!/usr/bin/env python3

import httplib2
import os
from apiclient import discovery
from memory_profiler import profile
from oauth2client import client, tools
from oauth2client.file import Storage
from time import sleep

SCOPES = "https://www.googleapis.com/auth/spreadsheets.readonly"
# See https://cloud.google.com/docs/authentication/getting-started
CLIENT_SECRET_FILE = ".client_secret.json"
APPLICATION_NAME = "ClientDebug"
DISCOVERY_URL = "https://sheets.googleapis.com/$discovery/rest?version=v4"

def get_credentials():
    home_dir = os.path.expanduser("~")
    credential_dir = os.path.join(home_dir, ".credentials")
    flags = None
    if not os.path.exists(credential_dir):
        os.makedirs(credential_dir)
    credential_path = os.path.join(credential_dir,
                                "sheets.googleapis.com-clientdebug.json")

    store = Storage(credential_path)
    credentials = store.get()
    if not credentials or credentials.invalid:
        flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
        flow.user_agent = APPLICATION_NAME
        credentials = tools.run_flow(flow, store, flags)

    return credentials

@profile(precision=4)
def get_responses(creds):
    """Fetch spreadsheet data."""
    sheet_id = "1TowKJrFVbT4Bfp-HFcMh_CZ5anfH0CLfmoqCz9SUr9c"

    http = creds.authorize(httplib2.Http())
    service = discovery.build("sheets", "v4", http=http,
                discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)
    result = service.spreadsheets().values().get(
            spreadsheetId=sheet_id, range="A1:O").execute()
    values = result.get("values", [])

    print("Got {} rows".format(len(values)))

if __name__ == "__main__":
    creds = get_credentials()
    for i in range(0, 50):
        get_responses(creds)
        sleep(2)

For measurements I used memory_profiler module with following results:
First and second iteration
Got 760 rows
Filename: ./main.py

Line #    Mem usage    Increment   Line Contents
================================================
    35  26.5195 MiB  26.5195 MiB   @profile(precision=4)
    36                             def get_responses(creds):
    37                                 """Fetch spreadsheet data."""
    38  26.5195 MiB   0.0000 MiB       sheet_id = "1TowKJrFVbT4Bfp-HFcMh_CZ5anfH0CLfmoqCz9SUr9c"
    39
    40  26.5195 MiB   0.0000 MiB       http = creds.authorize(httplib2.Http())
    41  26.5195 MiB   0.0000 MiB       service = discovery.build("sheets", "v4", http=http,
    42  29.2891 MiB   2.7695 MiB                   discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)
    43  49.5742 MiB  20.2852 MiB       result = service.spreadsheets().values().get(
    44  49.5742 MiB   0.0000 MiB               spreadsheetId=sheet_id, range="A1:O").execute()
    45  49.5742 MiB   0.0000 MiB       values = result.get("values", [])
    46
    47  49.5742 MiB   0.0000 MiB       print("Got {} rows".format(len(values)))

Got 760 rows
Filename: ./main.py

Line #    Mem usage    Increment   Line Contents
================================================
    35  49.5742 MiB  49.5742 MiB   @profile(precision=4)
    36                             def get_responses(creds):
    37                                 """Fetch spreadsheet data."""
    38  49.5742 MiB   0.0000 MiB       sheet_id = "1TowKJrFVbT4Bfp-HFcMh_CZ5anfH0CLfmoqCz9SUr9c"
    39
    40  49.5742 MiB   0.0000 MiB       http = creds.authorize(httplib2.Http())
    41  49.5742 MiB   0.0000 MiB       service = discovery.build("sheets", "v4", http=http,
    42  49.5742 MiB   0.0000 MiB                   discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)
    43  67.9922 MiB  18.4180 MiB       result = service.spreadsheets().values().get(
    44  67.9922 MiB   0.0000 MiB               spreadsheetId=sheet_id, range="A1:O").execute()
    45  67.9922 MiB   0.0000 MiB       values = result.get("values", [])
    46
    47  67.9922 MiB   0.0000 MiB       print("Got {} rows".format(len(values)))

Last iteration
Got 760 rows
Filename: ./main.py

Line #    Mem usage    Increment   Line Contents
================================================
    35 229.6055 MiB 229.6055 MiB   @profile(precision=4)
    36                             def get_responses(creds):
    37                                 """Fetch spreadsheet data."""
    38 229.6055 MiB   0.0000 MiB       sheet_id = "1TowKJrFVbT4Bfp-HFcMh_CZ5anfH0CLfmoqCz9SUr9c"
    39
    40 229.6055 MiB   0.0000 MiB       http = creds.authorize(httplib2.Http())
    41 229.6055 MiB   0.0000 MiB       service = discovery.build("sheets", "v4", http=http,
    42 229.6055 MiB   0.0000 MiB                   discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)
    43 229.6055 MiB   0.0000 MiB       result = service.spreadsheets().values().get(
    44 229.6055 MiB   0.0000 MiB               spreadsheetId=sheet_id, range="A1:O").execute()
    45 229.6055 MiB   0.0000 MiB       values = result.get("values", [])
    46
    47 229.6055 MiB   0.0000 MiB       print("Got {} rows".format(len(values)))

There's clearly a memory leak, as the reproducer fetches the same data over and over again, yet the memory consumption keeps rising. Full log can be found here.
As a temporary workaround for one of my long-running applications I use an explicit garbage collector call, which mitigates this issue, at least for now:
...
import gc
...
    result = service.spreadsheets().values().get(
            spreadsheetId=sheet_id, range="A1:O").execute()
    values = result.get("values", [])

    gc.collect()
...

I went a little deeper, and the main culprit seems to be in the createMethod function when creating dynamic method batchUpdate:
Method 'batchUpdate, approx. __doc__ size: 2886834
<class 'function'>
Filename: /home/fsumsal/venv/googleapiclient/lib64/python3.6/site-packages/googleapiclient/discovery.py

Line #    Mem usage    Increment   Line Contents
================================================
  1064     48.7 MiB     48.7 MiB     @profile
  1065                               def _add_basic_methods(self, resourceDesc, rootDesc, schema):
  1066                                 # If this is the root Resource, add a new_batch_http_request() method.
  1067     48.7 MiB      0.0 MiB       if resourceDesc == rootDesc:
...
  1086
  1087                                 # Add basic methods to Resource
  1088     48.7 MiB      0.0 MiB       if 'methods' in resourceDesc:
  1089     66.8 MiB      0.0 MiB         for methodName, methodDesc in six.iteritems(resourceDesc['methods']):
  1090     56.0 MiB      0.0 MiB           fixedMethodName, method = createMethod(
  1091     66.8 MiB     18.1 MiB               methodName, methodDesc, rootDesc, schema)
  1092     66.8 MiB      0.0 MiB           print(type(method))
  1093     66.8 MiB      0.0 MiB           self._set_dynamic_attr(fixedMethodName,
  1094     66.8 MiB      0.0 MiB                                  method.__get__(self, self.__class__))
  1095                                     # Add in _media methods. The functionality of the attached method will
  1096                                     # change when it sees that the method name ends in _media.
  1097     66.8 MiB      0.0 MiB           if methodDesc.get('supportsMediaDownload', False):
  1098                                       fixedMethodName, method = createMethod(
  1099                                           methodName + '_media', methodDesc, rootDesc, schema)
  1100                                       self._set_dynamic_attr(fixedMethodName,
  1101                                                              method.__get__(self, self.__class__))

(This method has a huge docstring.)
Nevertheless, there is probably a reference loop somewhere, as the gc.collect() call manages to collect all those unreachable objects.

I've been able to replicate this with a variant of your script, thanks!
I think it may be possible to cache the result of createMethod such that we don't create a new closure for every call, which would make the memory leak less dramatic (probably not noticeable), and faster.  But before I do that I'll try to identify the actual cycle.

Running debugging commentary for future reference:

Not a function of the docstring being large; if I change createMethod to not set the docstring onto the dynamically generated method, it still leaks.

Causing the generated method to be a noop doesn't fix the leak.

This is the "leak", a very simple circref.  It cannot be avoided without a redesign.

Now that we've figured why we perceive there is a leak, a) is it a problem?  and b), if so, what can we do about it?
Is it a problem?
Yes and no.
Theoretically, no, it's not a problem.  Reference cycles in programs are normal, and Python's garbage collector will eventually trash the objects involved in cycles.  Every time a Resource object is created (by calling methods on other Resource objects), we get some cycles between that Resource and its dynamic methods, and, in theory, this is fine.
Practically, yes, it's a problem. Repeated Resource creatoin causes the process RSS to bloat, and, on Linux at least, the memory consumed by these references is not given back to the OS due to memory fragmentation, even after the cycles are broken.
What can we do about it?
I've put it some work on a branch (https://github.com/mcdonc/google-api-python-client/tree/fix-createmethod-memleak-535)  trying to make the symptoms slightly better, by caching the functions that become methods on Resource objects, only creating one function per input instead of one per call.  This is not a reasonable fix, however, because refs involved in cycles still grow; every time a Resource is instantiated, it binds itself to some number of methods, and even if the functions representing these methods are not repeatedly created, the act of binding cached methods to each still creates cycles.  In order to truly fix the problem we would need to do one of these things:

Cache Resource instances themselves, instead of just caching the result of dynamic method creation.

Cause Resource instances to refer to any subresource they create and expose a "close()" method on resources, which would clear its dynamic methods and the dynamic methods of any subresource recursively, which would break the refcycle.

Generate code a different way; instead of instantiating Resource objects and binding methods to them the descriptor protocol, generate a class and all of its methods inline at the point where it is required.

Thank you very much for the thorough analysis. As you already said, a complete fix would require some time to implement, so the temporary fix using a cache along with gc.collect() is good enough for now at keeping the memory consumption at bay.
Again, thanks!

@mrc0mmand yes, in your particular case, creating "sheets" only once would make it leak so little that you won't need gc.collect()

@theacodes can you advise about which one of the options in https://github.com/google/google-api-python-client/issues/535#issuecomment-404994715 is most appropriate?

If I get a vote I'd like to see the second option, adding a .close() method. I've spent the past week or so tracking down this same memory error and found my way to this page. It happens I specifically looked for close() methods in the Resource objects because I knew something somewhere wasn't being released.
Adding a .close() method seems cleaner than my having to call gc.collect(). Either way I have to do something to cleanup resources and calling .close() is analogous to what we do already for files and other things.
In any case, this issue should be mentioned in the documentation and sample code, please!

I have a cron job, on google app engine, that reads data in from a google sheet. I am noticing the same memory leak (or maybe a different memory leak?). I tried the recommend work arounds: 1. creating the "sheets" object only once, and use gc.collect(). Neither worked in my case.  As a a test, I changed the few lines of code that read data from a google sheet to read data from a database table, and the memory leak went away.

I have a cron job, on google app engine, that reads data in from a google sheet. I am noticing the same memory leak (or maybe a different memory leak?). I tried the recommend work arounds: 1. creating the "sheets" object only once, and use gc.collect(). Neither worked in my case. As a a test, I changed the few lines of code that read data from a google sheet to read data from a database table, and the memory leak went away.

Can you help me to clarify your last sentence here?  Did you fix the code, or just confirm the memory leak?  I'm in the same situation, app engine job that reads google sheet and started getting "Exceeded soft memory limit" errors.

I never fixed it... in the short term, I used a high mem appengine instance so that would take longer to hit the memory threshold and then, as a long term solution, I switched to airtable instead of google sheets.

I never fixed it... in the short term, I used a high mem appengine instance so that would take longer to hit the memory threshold and then, as a long term solution, I switched to airtable instead of google sheets.

Got it - I'll look into using airtable instead.  Great suggestion.  Appreciate the help.

My solution in the end was just too use "pure" http requests

@AmosDinh Can you elaborate? In my project the memory leak issue has reared its ugly head again and I am looking for new approaches to deal with it.

@hx2A Well I don't use the the api at all,except for creating credentials. Here is an example from my class.
credentials = ServiceAccountCredentials.from_json_keyfile_name(
            "crypto-wscollector-1d1fc4f5c9ad.json",
            scopes=[
                'https://www.googleapis.com/auth/drive',
                'https://www.googleapis.com/auth/spreadsheets',

            ])
delegated_credentials = credentials.create_delegated(
            [service_account_email])
access_token = delegated_credentials.get_access_token().access_token
self.headers = {
            "Authorization": "Bearer "+access_token,
        }

and
def spreadsheetAppend(self,spreadsheetId,list2d):
        valueInputOption="RAW"
        range='!A:A'
        url = f"https://sheets.googleapis.com/v4/spreadsheets/{spreadsheetId}/values/{range}:append?valueInputOption={valueInputOption}"
        data = json.dumps({
            "range":range,
            "majorDimension":"ROWS",
            "values":list2d
        })
        r = requests.post(url,headers=self.headers,data=data)
        if str(r.status_code)[0] == "4":
            print(r,r.headers)

@AmosDinh Thank you, I understand now. This is helpful.

hx2A glad I could help you

@hx2A glad I could help you

Alternatively, you could have the sheets api code in another process which you could terminate after execution / RAM usage hitting a certain threshold. - Just wanted to include that option

@AmosDinh I'm trying to get this working now but I keep getting 403 'Forbidden' responses. I believe it has something to do with my service account's roles and permissions. Can you tell me about how your service account is configured? My current memory leaking code doesn't seem to be using the service account so I need to be sure it is configured correctly.

Which drive are you accessing? Your personal one, which you can reach by going to https://drive.google.com? In that case you have to add the service account by its email to a folder as editor/owner

I just got it to work. I had multiple problems but a big part of it was I needed to share the files on my gdrive with the service acount's email address. Thanks!

I had the same memory leak so I just sprinkled gc.collect() everywhere and bam now its manageable. I doubt that this would count as a fix though

@marvic2409 you might have a slow memory leak that leaks a small number of MB every hour. For a decent-sized system, it will take some time to become a problem.

Circling back to this, this recommendation from https://github.com/googleapis/google-api-python-client/issues/535#issuecomment-404994715 is the best way to avoid this.

Only creating "sheets" once prevents the leak too (if it's created more than
once, the leak remains):

creds = get_credentials()
http = creds.authorize(httplib2.Http())
service = discovery.build("sheets", "v4", http=http,
            discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)
sheets = service.spreadsheets()
for i in range(0, 50):
    get_responses(sheets)
    sleep(2) # no leak

Creating multiple service objects results in (1) potential memory problems and (2) takes extra time for refreshing credentials. If you're creating a service object inside a loop, or a function that's called more than once, move it outside the loop/function so it can be reused.


Ensure space supporter receives proper error code for specific droplet endpoints

Context
We are introducing a new role and we want to make sure it has the right access.
Expected result
A space should get a 403 unauthorized error when trying to curl the following endpoints:
PATCH /v3/droplets/:guid
DELETE /v3/droplets/:guid
POST /v3/droplets
POST /v3/droplets?source_guid=:guid
GET /v3/droplets/:guid/download
POST /v3/droplets/:guid/upload
Acceptance
As a space supporter, if I curl the above endpoints for a droplet that exists, I receive a 403 error telling me I am not authorized to perform this action.
Documentation
No changes required. We are just aligning with our style guide.

@sweinstein22 @monamohebbi should this issue include POST /v3/droplets? Currently we only see POST /v3/droplets?source_guid=:guid, which translates to copying a droplet. What about creating a droplet, is that something a space supporter can do?

I believe that was an oversight, according to the spreadsheet of endpoints they shouldn't be able to hit POST /v3/droplets. I'll add it to the description of the issue, thanks for checking in!

@sweinstein22 GET /v3/droplets/:guid/download seems like read-only permissions (ie. something that the space supporter should be able to do). In addition, we see that in the spreadsheet of endpoints, in the cf cli tab, download-droplet is listed as an allowed command.
❓ Why should GET /v3/droplets/:guid/download return 403 rather than succeed with a 302?

I believe this was mainly because we couldn't think of a use case where a space supporter persona would need to download a droplet. Thanks for sharing that it's read-only for all roles -- perhaps it would be fine to allow it even if it likely won't be used.

@piyalibanerjee @monamohebbi it seems possible (albeit bad practice) to store staging env vars in a droplet. Additionally, the droplet download would include the start command of an app which again could contain env vars (also bad practice).
🤔 Are those potential deal-breakers for allowing space supporters to download droplets?

Sorry for the delayed response.
It seems like the problem with returning a 302 is that for Cloud Foundry deployments with internal blobstores CC will return the the actual source code. Preventing the user from seeing source code is one of the main goals of the space supporter, so even though it is a small edge case I think we want to continue to return 403s for the download droplet endpoint.

@monamohebbi we updated the PR with 403 status for download droplet endpoint returned to space supported.


Rendering artifacts when panning an area chart

Expected Behavior
Area charts should render properly.
Current Behavior
If animations are disabled, and the zoom plugin is loaded, artifacts appear when panning out of the data range.
Initial Load:

After Panning:

Possible Solution
Not a solution, but I believe the issue to be in Chart.js, not the zoom plugin, as I've replicated the issue with my own custom zoom plugin.
Steps to Reproduce
Configure an area chart with a time x-axis (to allow panning past the range of data), and pan the data out of range.
https://jsfiddle.net/ke12g95r/5/

This also occurs with just a plain linear X axis. My gut is that the filler plugin is not correctly filtering points that are off the chart area, but it only seems to happen in certain scenarios

To me, looks like it draws to the first hidden point, but fails to go to origin from there, before closing the path. Could be simple fix or not :)

This behavior comes from #7793

guess it was a simple fix after all. thanks for the quick help!


OSVR-Unity Android app crashes when multithreaded rendering is enabled

Looking into it. Have to make sure the Unity context is current when calling RM functions.

multithreaded rendering on android now works on this branch: https://github.com/OSVR/OSVR-Unity/pull/245 with https://github.com/OSVR/OSVR-Unity-Rendering/tree/d3d_capi_db_refactor


Add food options to logistics sheet

...or tell me where to find it without sifting through waffle cards
I'm trying to build out the schedule, please add the catering and high level menu so I can add to the webpage
https://docs.google.com/spreadsheets/d/156IMrwWhpbRWy-TSvwplt6DNpCebgT7oV5MXU6iaa6o/edit#gid=4

Added.


Patch for wiki page spelling

From 232fc0f31b81f123e880f8cb56f2ac4a6912d8c1 Mon Sep 17 00:00:00 2001
From: Jeremy Wright<EMAIL_ADDRESS>Date: Wed, 17 Mar 2021 10:11:08 -0700
Subject: [PATCH] Fix a few misspellings.

---
 Getting-Started.md        | 4 ++--
 Setting-up-Environment.md | 2 +-
 Settings.md               | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/Getting-Started.md b/Getting-Started.md
index b8dbfec..d3e7d72 100644
--- a/Getting-Started.md
+++ b/Getting-Started.md
@@ -38,8 +38,8 @@ Run command `TLA+: Parse module`. It will translate the PlusCal algorithm to a T

 We now have a simple specification that we can check by running the command `TLA+: Check model`. It will start the TLC tool on the currently open specification and display its progress and final result in a special panel.

-One of the artefacts that the TLC command cerates when running on a `.tla` file with a PlusCal algorithm is a `.cfg` file that contains the model parameters. If you don't use PlusCal in your specification, the model configuration file will not be created automatically, but the extension will warn you about its absence and propose you to create it from a simple template.
+One of the artifacts that the TLC command creates when running on a `.tla` file with a PlusCal algorithm is a `.cfg` file that contains the model parameters. If you don't use PlusCal in your specification, the model configuration file will not be created automatically, but the extension will warn you about its absence and propose you to create it from a simple template.

 You can find the full output of the TLC tool in a `.out` file that will be created near your specification. Should you need to visualize an output from a previous model checking, use the command `TLA+: Visualize TLC output` on a `.out` file.

-Should you have any problems while checking the TLA+ module or running TLC, please, refer to the [Troubleshooting](/alygin/vscode-tlaplus/wiki/Troubleshooting) page, it might provide a solution.
\ No newline at end of file
+Should you have any problems while checking the TLA+ module or running TLC, please, refer to the [Troubleshooting](/alygin/vscode-tlaplus/wiki/Troubleshooting) page, it might provide a solution.
diff --git a/Setting-up-Environment.md b/Setting-up-Environment.md
index e3535ee..a3867b7 100644
--- a/Setting-up-Environment.md
+++ b/Setting-up-Environment.md
@@ -4,5 +4,5 @@ If you have already [installed](/alygin/vscode-tlaplus/wiki/How-to-Install) the
 - Configure [automatic module parsing](/alygin/vscode-tlaplus/wiki/Automatic-Module-Parsing)
 - Adjust [formatting options](/alygin/vscode-tlaplus/wiki/Formatting-Preferences) to your liking
 - Improve [visualization of your specifications](/alygin/vscode-tlaplus/wiki/Fonts)
-- Install the [vscode-pdf](https://marketplace.visualstudio.com/items?itemName=tomoki1207.pdf) extension to preview specificaion PDFs
+- Install the [vscode-pdf](https://marketplace.visualstudio.com/items?itemName=tomoki1207.pdf) extension to preview specification PDFs
 - Learn how to [visualize generated states](/alygin/vscode-tlaplus/wiki/Visualizing-States)
diff --git a/Settings.md b/Settings.md
index 08da067..490d75a 100644
--- a/Settings.md
+++ b/Settings.md
@@ -14,7 +14,7 @@ These settings are available in the Extensions → TLA+ section of the [Setti

 ## Other Useful Settings

-These settings and configuration tricks are not provided by the extension itself, but leverage the general VS Code platform capabilibies:
+These settings and configuration tricks are not provided by the extension itself, but leverage the general VS Code platform capabilities:

 * [Keyboard Shortcuts](/alygin/vscode-tlaplus/wiki/Keyboard-Shortcuts)
 * [Automatic Module Parsing](/alygin/vscode-tlaplus/wiki/Automatic-Module-Parsing)
--
2.24.3 (Apple Git-128)

Thank you for the wonderful Extension!

@JeremyLWright, thanks for the patch! Applied.


CME in WireTracker

https://pastebin.com/JG3wKGAQ

How frequently does this occur?
Any plugins that utilize LeadWires API?

It does happen for me also sometimes. But really not often. Maybe once in a Week. But always without any connection to other messages in the logs. Sometimes the server lagged a few minutes before or a player joined but not directly after or before anything that is logged happened.
And no Plugin utilizes the API.

More often (about every two days, sometimes multiple times a day) I also get this warning in the console: https://pastebin.com/3iJ65re6 It appears mostly a few minutes (occasionally seconds) after someone joined or left the server. It's both nothing serious I guess but I would appreciate it if it would be fixed :)

Hi, I've created a separate issue: https://github.com/saharNooby/lead-wires/issues/4
I think it can be fixed pretty easily, but please write your spigot version info here


Fix some portuguese translations

Commit to fix some portuguese translations that were wrong or missing.

Great job @guiiruiz, thank you!


Sunsynk library docs/example code

Issue related to the Python sunsynk library
I've just found this, and it looks perfect for my Sunsynk 8kw inverter.
However I'm really struggling to find any documentation on the library other than pip install sunsynk. Is there anything available?
Ideally, I'd really like just a simple 'hello world' app - a command-line tool that displayed one or two of the inverter's registers.
Once I had that I could dig into the code enough to figure out whatever else I needed, but it would also be a hugely helpful debugging aid. I've just made up the cable and I'd like to see if it was working before I have to worry about whether Home Assistant is set up correctly.
Thanks - and sorry if this isn't the best way to get in touch.

Hi, the sunsynk library is used extensively by the homeassistant addon, in the same repo:
Init the driver and then simply read values with await read_sensors(list_of_sensors)
You can see this library as a wrapper (with sunsynk register info) over either one of the two modbus drivers: umodbus or pymodbus

Thanks for the speedy response! I just tried this, but sadly no luck:
import asyncio
import pprint
from sunsynk.definitions import ALL_SENSORS, DEPRECATED
from sunsynk.sunsynk import Sensor, Sunsynk
from sunsynk.pysunsynk import pySunsynk
from sunsynk.usunsynk import uSunsynk

#pprint.pp(ALL_SENSORS)
SERIAL = ALL_SENSORS["serial"]
sensors = [ALL_SENSORS["pv1_power"], ALL_SENSORS["pv2_power"]]

SUNSYNK = uSunsynk()
#SUNSYNK = pySunsynk()
SUNSYNK.port = "serial:///dev/ttyUSB0"
#SUNSYNK.server_id = 1
#SUNSYNK.timeout = 10
#SUNSYNK.read_sensors_batch_size = 60

async def main():
  print("Connecting...");
  await SUNSYNK.connect()
  print("Connected. Reading...")
  await asyncio.wait_for(SUNSYNK.read_sensors([SERIAL]), 10)
  print("Serial "+SERIAL.value)
  #print("Reading 2...")
  #await asyncio.wait_for(SUNSYNK.read_sensors(sensors), 10)
  #print(sensors[0].value)
  #print(sensors[1].value)
  print("Finished");

LOOP = asyncio.get_event_loop()
LOOP.set_debug(True)
LOOP.run_until_complete(main())
LOOP.close()

It gives me a timeout:
Connecting...
Connected. Reading...
Traceback (most recent call last):
  File "test.py", line 34, in <module>
    LOOP.run_until_complete(main())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "test.py", line 23, in main
    await asyncio.wait_for(SUNSYNK.read_sensors([SERIAL]), 10)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

It's a SunSynk 8,8kW, wired to the RS485 connector, with the wiring and USB adaptor as shown (using pins 1+2) in the pics on https://github.com/kellerza/sunsynk#hardware (there's no GND since the adaptor doesn't have it).
Does it look like the code I've got there is ok? Is there anything else I need to set up in the inverter to enable RS485?
If/when I finally get this going, it is worth me doing a quick PR for it?

Hi, I would simpy change the read to  await SUNSYNK.read_sensors(sensors)
So your main method would be:
async def main():
  print("Connecting...");
  await SUNSYNK.connect()
  print("Connected. Reading...")
  await SUNSYNK.read_sensors(sensors)
  print(", ".join(f"{s.name}={s.value} {s.unit}" for s in sensors))
  print("Finished");

Thanks! And thank you for the documentation update on the Server ID. Mine was set to 00 by default. I set it to 01 and uncommented SUNSYNK.server_id = 1 and that fixed it for me!
Out of interest, does it matter if the inverter is set as a master or slave?
I'll close this issue, but do you want me to add a PR for the script with the changes you suggested (or just feel free to copy it in yourself if it's easier)? I guess sunsynk/scripts/comms-test.py or similar, and a mention in the README?

PRs are always welcome, you can add comms-example.py
The master/slave setting is only used for a multi-inverter setup. From my understanding is is not related to or affects the modbus master/slave client/server in any way


fix for failed to get system container stats errors

Signed-off-by: Parul<EMAIL_ADDRESS>
Which issue(s) this PR addresses:

Closes #475
This PR adds  "/system.slice/crio.service" to kubelet flags to suppress error message regarding
"Failed to get system container stats" err="failed to get cgroup stats for \"/system.slice/crio.service\": failed to get container info for \"/system.slice/crio.service\": unknown container \"/system.slice/crio.service\"" containerName="/system.slice/crio.service"

/lgtm


Page masthead image above/next to/instead of site title?

Minima is a gorgeously simple theme, but given most sites have some sort of logo or masthead image, I would think allowing for an (optional) masthead image to go with the site title would be an addition that doesn't yet constitute feature-creep. I'm thinking of something where you specify a masthead-image, masthead-left-image or masthead-above-image in the _config.yml.
Would anyone be interested in adding this? If I managed to somehow add this feature to my copy of Minima, would there be interest in merging that back into the mainline?

Okay, I managed to build this. In _includes/header.html, replace the existing title output code with:
	{%- if site.masthead -%}

	{%- else -%}
	{{ site.title | escape }}
	{%- endif -%}

In _sass/minima/_layout.scss, add the following CSS class:
.site-masthead {
  display: block;
  margin-left: auto;
  margin-right: auto;
  padding-top: $spacing-unit / 2;
  padding-bottom: $spacing-unit / 2;
}

And in _config.yml add:
# Specify the name of a file to show as a masthead graphic at the top of each page, instead of a title, here:
# masthead: /my_masthead.jpg

Would be grateful if someone could put that into the official Gem.

Would you mind opening a PR so that we can test this out?

See PR #266.


Map.setMilStdLabels is not working

engineInterface.map.config is sending over just the messageId and no settings.

To test:
-activate the map set milstdlabels from the tester
-verify toaster popup states the labels have been changed to the selected label
-verify on the map engine that the labels have changed

Fixed in core and in cesium. To test try changing the labels on a satellite. Apparently this intentionally does not toggle labels for multipoint graphics

Reintegrate

Closed #88


Incorrectly validates and formats UK

In the uk you type your mobile number with a 0 at the start, if you do this it doesn't recognise and format the number

Incidentally it works on this version: https://ry3wlvlxkn.codesandbox.io/


App should detect if cannot reach API network

Use case is publicly hosted "App", trying to reach the API server in private VPC / requiring VPN.
Currently, it would be a network timeout, and might log errors to console.log().  We should make sure there's a user-friendly error message that explains that the API server is not available.

The trick will be to detect no network connectivity.  That's probably a special error from fetch.
When detected, we should show an error with the Mantine Notification system: https://mantine.dev/others/notifications/
We can use autoClose: false to keep the notification open.


Using Carbon for dates in PHP 8

Proposed title of article
[Languages]Using Carbon for dates in PHP 8
Use keyword research to improve your article's SEO.
Proposed article introduction
Carbon is a PHP package developed by Brian Nesbit which prolongs PHP’s DateTime class. It offers some of the excellent functionality to handle with dates in PHP; It primarily deals with date timezones.
You will learn to add minutes on a current date with the help of Carbon package in Laravel 8, and this tutorial will demonstrate the Carbon in Laravel and specifically show you how to use addMinute(), addMinutes(), subMinutes() methods using Carbon date object.
In this article, you will install Carbon and explore the features and functionality that it provides.
Key takeaways

Define Minute with Laravel Carbon
Define Sub Minutes with Carbon
Getting a Specific Date and Time

Kindly, we only accept one topic at a time at the moment. Thanks

Hi, kindly reopen this, am ready to work on it. Thank you!


Exporting data as markdown instead of HTML or JSON

Is it possible to use tiptap to send the raw markdown instead of HTML or JSON?

Duplicate of #66


Updated version v4.1.5 with Citrus Wallet Rebranding.

Changed the launcher icon.

Release version 4.1.5


[network-key] Change thrNetworkKey for docs

@gavinmcgovern , need to also increment:
https://github.com/openthread/openthread/blob/05fd1deb497451464ff050102d8f2994e53e6820/include/openthread/instance.h#L56


Gigasecond not working with Groovy 2.4.5?

I am very new to Groovy and thought about learning it with Exercism.io. I have set-up an environment and did the first two exercises. The third one however seems not working.
I thought it was an error on my side or a bug in the groovy.time.TimeCategory module. So I looked at the example solution. Well, it is not working either:
Test Failure: gigasecondFromDateWithHoursAndMinutes(GigasecondTest) java.lang.AssertionError: expected:<Wed Mar 27 13:17:40 CET 1991> but was:<Wed Mar 27 14:17:40 CET 1991>
Test Failure: gigasecondFromDateWithHoursAndMinutesAndSeconds(GigasecondTest) java.lang.AssertionError: expected:<Thu Feb 19 03:02:25 CET 2009> but was:<Thu Feb 19 04:02:25 CET 2009>
There is one hour difference in these two tests. The third test works.
I run the tests on Groovy Version: 2.4.5 JVM: 1.8.0_05 Vendor: Oracle Corporation OS: Windows 7.
Can anyone verify the error or tell me what I do wrong?

It's possible you aren't doing anything wrong— I don't remember the specific code which gave me problems, but I did run into some edge problems with the tests (I believe on Gigasecond) when working on retrofitting them as Spock specifications.
I don't know what the appetite is for the retrofit, but it seems to have some positive response.  If it doesn't, I can cycle back and try to firm up the existing Gigasecond tests.

I have taken a look at your Gigasecond Spec. The expected values are the same as previously so it differs from what I get when I execute my code (or the example solution). And my difference is still +1 hour. Even with Groovy 2.4.7. Even when executing the code as a simple Groovy script.
The interesting is if I change the JVM property user.timezone from my current (CET) to UTC or GMT I get 3 test failures not just the two mentioned above.
However when I change the timezone to PST or CST it works fine. Can there be some time-calculation difference between these timezones which we are not aware of? Or can it be a Groovy issue?

Well, I am closing this issue because it seems like a Groovy one and not related to exercism.


Access on Konga

Hi everybody,
by carrying out the ldap configuration, functioning correctly without anomalies, it is no longer possible to access the UI via the Admin user which is created in the first instance.
is it intended/normal, or can it be bypassed in some way?
one possible way would be to first call the DB (or the file in case of non-db) and query on it.

If the user exists on the db and is a Local user, I login with that.
If the user does not exist on the db, I call the LDAP for the request and save the data on the db.

Thank you very much,
Greetings

.


GridSearchCV error when n_jobs=-1

Hello,
When I use GridSearchCV using njob=-1 I got an error bellow but it work great with njob=1. When using njob=-1 it runs for the first argument of taille_fen (7) but got error for the next. But it work with njob=1...
The above exception was the direct cause of the following exception:
PicklingError                             Traceback (most recent call last)
<ipython-input-17-692078694243> in <module>()
     33     grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=tscv, n_jobs=-1, verbose=3)
     34
---> 35     res = grid.fit(x_train, y_train,callbacks=[es])
     36     grid_result.append(res)
     37

7 frames
/usr/lib/python3.7/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

PicklingError: Could not pickle the task to send it to the workers.

My code is
from keras.callbacks import EarlyStopping
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import KFold, TimeSeriesSplit, GridSearchCV
import pickle
from google.colab import files

dim_LSTM = [16,32,64,128]
l1_reg = [0,0.001,0.01,0.1]
l2_reg = [0,0.001,0.01,0.1]
taille_fen = [7,10,15,30,50]
bs = [512]
lrate = [0.1]

max_periodes = 1

es = EarlyStopping(monitor='loss', mode='min', verbose=1, patience=20, min_delta=1e-4, restore_best_weights=True)
tscv = TimeSeriesSplit(n_splits = 5)

grid_result = []
for bsize in bs:
  for taille in taille_fen:
    param_grid = {'dim_LSTM': dim_LSTM, 'l1_reg': l1_reg, 'l2_reg': l2_reg, 'lrate': lrate, 'taille_fen': [taille], 'bs':[bsize]}

    dataset = prepare_dataset_XY(serie_entrainement,taille,horizon,bsize)

    x,y = tuple(zip(*dataset))
    x_train = np.asarray(tf.reshape(np.asarray(x,dtype=np.float32),shape=(np.asarray(x).shape[0]*np.asarray(x).shape[1],taille,1)))
    y_train = np.asarray(tf.reshape(np.asarray(y,dtype=np.float32),shape=(np.asarray(y).shape[0]*np.asarray(y).shape[1])))

    model = KerasRegressor(build_fn=ModelLSTM, epochs=max_periodes, verbose=2)
    grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=tscv, n_jobs=-1, verbose=3)

    res = grid.fit(x_train, y_train,callbacks=[es])
    grid_result.append(res)

Full error
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/joblib/externals/loky/backend/queues.py", line 153, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/usr/local/lib/python3.7/dist-packages/joblib/externals/loky/backend/reduction.py", line 271, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/usr/local/lib/python3.7/dist-packages/joblib/externals/loky/backend/reduction.py", line 264, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/usr/local/lib/python3.7/dist-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/usr/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 884, in _batch_setitems
    save(k)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1087, in save_functor
    obj.keywords), obj=obj)
  File "/usr/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1087, in save_functor
    obj.keywords), obj=obj)
  File "/usr/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/usr/local/lib/python3.7/dist-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 682, in _save_reduce_pickle5
    dictitems=dictitems, obj=obj
  File "/usr/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1177, in save_cell
    f = obj.cell_contents
ValueError: Cell is empty
"""

The above exception was the direct cause of the following exception:

PicklingError                             Traceback (most recent call last)
<ipython-input-17-692078694243> in <module>()
     33     grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=tscv, n_jobs=-1, verbose=3)
     34
---> 35     res = grid.fit(x_train, y_train,callbacks=[es])
     36     grid_result.append(res)
     37

7 frames
/usr/lib/python3.7/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

PicklingError: Could not pickle the task to send it to the workers.

@AlexandreBourrieau
This is not Build/Installation or Bug/Performance issue. Please post this kind of support questions at Stackoverflow. There is a big community to support and learn from your questions. GitHub is mainly for addressing bugs in installation and performance. Thanks

Nodofy have an idea ? IOn Stackoverflow nobody find the solution... Thanks !

@AlexandreBourrieau
Please find these link1, links2, hope it helps.Thanks

Thanks I finnaly found that by default the backend used is loky wich cannot share memory content between process so I need tio use threading backend to share memory content between thread.

You used cv=tscv  in the code what are the values of tscv? did you setup?


⚠ Bofrost.es has degraded performance

In 047343f, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 3279 ms

Resolved: Bofrost.es performance has improved in 6ba90cd.


Gateway contracts feedbacks

Removed value token (Please double check the usage)
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20CoGateway.sol#L216

Rename redeemProcess -> redeem
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20CoGateway.sol#L722

Remove the nonce parameter from EIP20 Gateway stake and EIP20CoGateway redeem method [Not 0.0.10 release]

Allow zero amount in increaseSupply. Also, Check UT if allows zero transfer.
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20CoGateway.sol#L1139

Try to make events specific to message bus states. The intention is to reduce the number of event types from the gateway.
[Not 0.0.10 release]

Don't return penalty in case of progress with proof. Burn the penalty. Check progressStakeWithProof and progressRedeemWithProof

https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20CoGateway.sol#L569
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20Gateway.sol#L500

Move deletion up
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20Gateway.sol#L1011

https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20CoGateway.sol#L451

Emit event on activating and deactivate Gateway
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20Gateway.sol#L1046

Move check that reward > stake/mint amount in confirmStakeIntent and confirmRedeemIntent
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/EIP20Gateway.sol#L1219

require(
            rewardAmount_ < redeemAmount_,
            "Reward amount must be less than redeem amount."
        );

Make comment on general in gateway base. Stake and redeem terms shouldn't be used in gateway base.
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/GatewayBase.sol#L73

Change bounty unlock time to 7 days as per aux block time.

https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/GatewayBase.sol#L79

Add circular buffer for storage roots.
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/GatewayBase.sol#L121

Improve variable names  in feeAmount method.
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/GatewayBase.sol#L461

14 . Gateway nonce should start with 0.
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/GatewayBase.sol#L686

In OST prime use  coin for base token(eth) and OST for token.
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/OSTPrime.sol#L58

Use co-gateway address explicitly

https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/OSTPrime.sol#L256

Use valueToken instead of token accross contracts(OSTPrime, UtilityToken, EIP20CoGateway, EIP20Gateway and SimpleStake)
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/UtilityToken.sol#L79

Change EIP20Interface to address
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/gateway/UtilityToken.sol#L79

Correct comment Inbox of source chain -> Target chain

https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/lib/MessageBus.sol#L245

Rename messageStatus to inboxMessageStatus

https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/lib/MessageBus.sol#L256

Remove dead code
https://github.com/OpenSTFoundation/mosaic-contracts/blob/fa436b06cacf3ea84d72550b7398f94220476648/contracts/lib/MessageBus.sol#L643

Clean bytes lib. Only keep the necessary methods.

@gulshanvasnani is it correct that I am assigned to this ticket?

@schemar , This ticket has been self-assigned by you.
Then I assigned it to myself because i was doing Change bounty unlock time to 7 days as per aux block time. task.

keeping this ticket around for later review but all contract changes have been closed, barring severe security issues that would rise

New task for this issue: review which changes have not been made, and create a new ticket listing only those, please.

Converting to an epic to add sub-tasks.

Removed value token

#692

Rename redeemProcess -> redeem

#693

Remove the nonce parameter from EIP20 Gateway stake and EIP20CoGateway redeem method [Not 0.10 release]

#694

Allow zero amount in increaseSupply. Also, Check UT if allows zero transfer.

#695

Try to make events specific to message bus states. The intention is to reduce the number of event types from the gateway. [Not 0.10 release]

#696

~Don't return penalty in case of progress with proof. Burn the penalty. Check progressStakeWithProof and progressRedeemWithProof (#668)~
~Move deletion up (#663)~
Emit event on activating and deactivate Gateway

#697

~Move check that reward > stake/mint amount in confirmStakeIntent and confirmRedeemIntent (#666 😈)~
~Make comment on general in gateway base. Stake and redeem terms shouldn't be used in gateway base (#690)~
~Change bounty unlock time to 7 days as per aux block time. (#671)~
Add circular buffer for storage roots.

#698

Improve variable names  in feeAmount method.

#699

Gateway nonce should start with 0.

#700

~In OST prime use  coin for base token(eth) and OST for token. #686~
~Use co-gateway address explicitly #674~
Use valueToken instead of token accross contracts(OSTPrime, UtilityToken, EIP20CoGateway, EIP20Gateway and SimpleStake)

#701

Change EIP20Interface to address

#702

~Correct comment Inbox of source chain -> Target chain #688~
Rename messageStatus to inboxMessageStatus

#703

~Remove dead code #664
~Clean bytes lib. Only keep the necessary methods. #664 and #675~

Created new tickets for the remaining tasks. This is now considered an Epic. @benjaminbollen please check the new tickets.

All tickets in the epic are closed. Closing this epic.


Bump version and create release notes for <IP_ADDRESS>

Signed-off-by: Craig Perkins<EMAIL_ADDRESS>Description
Create release notes and bump version to <IP_ADDRESS>

Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Maintenance
Check List

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@cwperks The Whitesource check failed. Looks like there're several new high CVEs.


Broken build due to reformatting

#109 (and others) seem to break the build.
During reformatting a lot of this. statements have been removed, leaving ambiguous symbols behind:
error CS0136: A local variable named 'exp' cannot be declared in this scope because it would give a different meaning to 'exp', which is already used in a 'parent or current' scope to denote something else
I'm compiling with VS2013.

@sharwell does this build with VS2015?

Yes, it does.

I think the issue is limited to cases where the semantics would actually cause a build error. By that I mean either the semantics are correct in 2013 builds, or you see a CS0136 error. I'll correct these errors and everything will be :smile: again.

I wonder why they have removed CS0136 in 2015 - now writing confusing code is allowed again?


docs(node-agent): add proposal

LGTM. Please squash your commits into 2 of them (1 for the readme+workload files, another one for the proto file) and we'll merge it.

Done! It should be ready to be merged. :+1: @sameo

Sorry for the new commit, after a discussion with the scheduler team, we figured the id was useless. The name is the unique identifier for a workload.

Sorry for the new commit, after a discussion with the scheduler team, we figured the id was useless. The name is the unique identifier for a workload.

No worries. Please squash your commit into the gRPC proto one, address my one last comment and we'll merge it.

done !


you need tests

As the number of commands increse, so does the need for testing.
Write some basic unit tests for each command as a starting point;
then add more tests as more bugs and features come in.

This is kind of "low prio"; since I'll be actively using sif I will be testing it "all" "the" "time" by definition.
a la, testing by dogfooding.


discuss: change CLINVAR id from RCV to Variant ID

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 96.831%

Totals

Change from base Build 972320854:
0.0%

Covered Lines:
439

Relevant Lines:
444

💛  - Coveralls

@colleenXu what's the justification to switch to using ClinVar variant_id instead of RCV id? Is that the primary ID for Clinvar in Biolink model? Just need a description to document why we made this change.
I also suggest to remove dbnsfp.clinvar.clinvar_id, clinvar field should be the canonical source for ClinVar.

@newgene It's not clear to me which ID the biolink-model is referring to with the CLINVAR prefix. See the id_prefixes here (wait...there's CLINVAR and ClinVarVariant? What are these?).
Based on my reading on Clinvar identifiers and this FAQ on RCV, it seemed that RCV were more like an association's ID (phenotype/disease + variant).
So I thought maybe a different ID from clinvar refers to the SequenceVariant itself and should be used...
According to the linked Clinvar doc on identifiers, the closest to a "unique variant ID" is maybe a "VariationID" or "AlleleID"?

OK, if this does not link to a known issue or expected use case, I would hold off this one.
We are making changes to ClinVar source at MyVariant.info now: https://github.com/biothings/myvariant.info/issues/127. Will reevaluate this after this myvariant issue is closed.

Based on the conversation in NCATSTranslator Slack, the CLINVAR ID prefix for SequenceVariant refers to the "Variation ID" in Clinvar.
My recommendation is to go forward with this PR, once I add a commit to remove the dbnsfp field mapping...
I've confirmed that the clinvar.variant_id field in MyVariant corresponds to the Clinvar Variation ID. For example, see how the clinvar.variant_id:596318 corresponds to the record here that states that the Clinvar Variation ID is also 596318


refactor!: Initialization screens

@amanji Able to add a description to describe the breaking ! part of the change. Ty?


Update README.md

Add link to container repository.

Pull Request Test Coverage Report for Build 884

0 of 0   changed or added relevant lines in 0 files are covered.
6 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.3%) to 90.479%

Files with Coverage Reduction
New Missed Lines
%

pkg/sm/sm.go
2
77.55%

storage/postgres/storage.go
4
85.71%

Totals

Change from base Build 870:
-0.3%

Covered Lines:
1796

Relevant Lines:
1985

💛  - Coveralls


add functionality for fed data downloads

add login for fed downloads
add possibility to save original download from fed

@pauleckhardt Would you like someone to review this? If so, please add a description of the PR on what it does and why, and optionally assign a reviewer.