Any way to import all historical data?

It seems the DB is populated from the time of install?
Any way to pull in all the history?

Hi @mikkyo9  - good question.  The Powerwall doesn't keep that history itself.  When you install the "Powerwall-Dashboard" stack, the Telegraf service starts polling the Powerwall to get the metrics.  The data is then stored in the DB for Grafana to display.  The Tesla App has this historical data because it (Tesla Cloud) started polling the Powerwall when it was commissioned.   I'm not aware of a way to pull this historical data into the database and even if we could, it would be missing some data like the String and Temperature data.

You could probably manually download each day's data and then write a script to transfer that information into the influx DB. I wish I could do an aggregate data download with fine granularity. Best I've found is to do a daily download in the phone app, then upload that to my dropbox and then view the info on my desktop.

@jasonacox - Your abilities greatly outpace mine, so I'm bringing an idea without specific testing. On https://www.teslaapi.io/powerwalls/state-and-settings and https://www.teslaapi.io/energy-sites/state-and-settings there appears to be an API call for historical data using a query period of day, week, month, year (similar to the granularity offered in the app). My thoughts are, could one develop a script with a start date, and then increment by day to download historical data?

Thanks @dmayo305 - I have been looking at those Tesla cloud APIs (used by the Tesla app) to add features to pyPowerwall  (e.g. update battery reserve setting). However, it seems Tesla keeps making breaking changes so I haven't found a good example/library to use TeslaPy seems to be close but I haven't been able to get it to work yet.  If we can figure it out, it wouldn't be too difficult to do what you mention.

I have a sample project that uses TeslaPy to download Powerwall and Solar site data as json files. I imagine it could be an example of how to get the data for adding to the Dashboard, but I don't know if it would need to be reformatted before loading in to the database.
This example uses the "refresh token" method for logging in, so you'll need to generate that first with a 3rd party app or one of the other examples floating around.

https://github.com/fracai/tesla-history
The sample project in question

Thanks @fracai !  Nice work.  Yes I think we could use that to port the data into InfluxDB.  What method did you use to get the "refresh token"?   If we extend your project to drop that data into the Dashboard DB, we could include instruction on how to make it work.
Ideally, I want to find an automated way to get the token so I can include that in the pyPowerwall library.

I used "Auth for Tesla", an iOS app. There are a few apps listed on this TeslaMate FAQ.
I think an automated method is ideal, but I've gotten the impression that manually retrieving the refresh token is the best method as Tesla frequently requires entering a captcha in order to login.
Fortunately, I think the refresh token is valid for a very long time and, unless you need to fill in data gaps, it shouldn't be necessary to request historical data. It might not be ideal, but I think it'd be acceptable to use a manual process to fill in old data.
Then again, if the refresh token is stored, it'd be possible to make a daily request of historical data to regularly fill in any missing gaps.

There's also this python script for retrieving the token, though it does require manual intervention.
https://github.com/bntan/tesla-tokens

Any more work being done for historical data? It'd be really cool to pull in all the old data.

I agree. It's not an easy fix, unfortunatelyl.  I'm doing more investigation and open to any help. ;)

What I am most interested in is my NEM year data(which started a month ago). So even if I could just import the basic data(to grid, from grid, home usage) as a single point in history it would be useful to fill in the gap from when I started to use powerwall-dashboard.

I have some good news regarding this! 😀
Thanks to @jasonacox for pointing out the TeslaPy module and also @fracai for the sample project using that. I started testing the module and have worked out how to import historical data via the Tesla Owner API (cloud) directly into InfluxDB. 🎉
Having >1yr of historical data that I would certainly love to see in my dashboard was a good motivator. 😊
I'm still working on the program but hopefully will be ready to release in the next week or few. Full time day job doesn't help 😞 - also I've had to brush up on my python knowledge, as these days I'm primarily coding in a language probably most people have never heard of!
So, I have developed this as a command line tool that accepts a start & end datetime range to retrieve history from the Tesla Owner API and import into InfluxDB. @jasonacox I'm thinking submitting a PR to add it to the tools directory seems appropriate?
I'm using the staged authorization process of TeslaPy as it seemed convenient and easy in my opinion. The login to your Tesla account is only required once, as after login a token is cached/saved and it appears to never expire (similar to the Tesla app which stays logged in).
Obviously not all data that the Powerwall-Dashboard logs is available for import. But, the main data shown in the "Energy Usage" and "Power Meters" panels is.
I think I can get the "Monthly Analysis" data working too (still testing). This is stored in the kwh retention policy, but, it is populated from a CQ based on the data in autogen. So I just need to run a query to refresh it after importing history data.
Also, I've been able to extrapolate the grid status from the backup event history, which is nice. 😃
In case there are problems with an import, obviously backups are highly recommended. But, the program will also tag all the data points with a "source=ownerapi" tag in InfluxDB. This makes it easy to reverse what was imported with a query like DELETE WHERE source='ownerapi' if something went wrong (and could be further limited to a specific time period if required). This would only delete the imported data, and not affect data that came from your gateway via the dashboard. Also note original dashboard data is never overwritten or changed if you were to enter a datetime range that overlapped with the dashboard data (although I would not recommend - it is best to just fill periods of missing data only).
The history data available from the Tesla cloud is in 5 minute intervals only (15 min for charge percentage), but it seems good enough. Here are some example comparisons.
Below is showing data logged from Powerwall-Dashboard as normal (i.e. every 5 seconds or so):

Compared with below where data was imported from the Tesla cloud - the graph and values (e.g. Power Meters) are pretty close:

Example showing imported history data with a backup event - the grid status value is extrapolated from backup event history:

If you have a data gap / missing data your graph might look like below:

This can fixed now by importing the history from Tesla cloud for the specific time period:

So it seems to be working well so far as I can tell, and useful for importing old data from before you installed Powerwall-Dashboard, as well as if your system went down for some reason!
While I'm still working on this, if anyone has any feature requests, questions, tips etc. please send them my way. 😊
(side note: I'm already thinking about fetching historical weather data as well now, which is available from the OpenWeather One Call API 3.0 "timemachine" API call 😉 )

Hi @mcbirse  Great work here!!!  Please feel free to submit a PR in the /tools directory.  I'm sure others would love this too.

I'm looking forward to testing this. I have several small data dropouts as well as several months of data before I got the dashboard setup.

Same here. Take your time to get it where you're happy, but I'm really looking forward to getting a look as well.
Thanks for working on it.

Update finally - I have been working on the script to import historical data from Tesla cloud over the past few weeks, and believe it is now good to go! :crossed_fingers:
I will submit a PR to add it to the /tools directory shortly.
Hoping for some feedback from the community on this, as so far all testing has been with my system only.
Please see the below instructions for how to use the script (will also add this to the tools README).
The tool can be used to import historical data from Tesla cloud, for instance from before you started using Powerwall-Dashboard. It can also be used to fill in missing data (data gaps) for periods where Powerwall-Dashboard stopped working (e.g. system down, or lost communication with the Local Gateway). By default, the script will search InfluxDB for data gaps and fill gaps only, so you do not need to identify time periods where you have missing data, as this will be done automatically.
NOTE:

The available data from Tesla cloud is limited to 5 minute intervals of power usage, and backup events only. It will not be as accurate as data logged by Powerwall-Dashboard, however is still sufficient to provide approximate usage history. The historical data is obtained via the same API as used by the Tesla mobile app, so should be very similar in accuracy to what is displayed in the app.
Data imported by this tool will not overwrite existing data, and will update the following panels only: Energy Usage, Grid Status, Power Meters, and Monthly Analysis

To use the script:

Install the required python modules: pip install python-dateutil teslapy influxdb
Follow the steps below

Setup and logging in to Tesla account
On first use, it is recommended to use the --login option. This will create the config file, save an auth token so you will not need to login again, and then display the energy site details associated with your Tesla account.
# Login to Tesla account
python3 tesla-history.py --login

It will run in an interactive mode. The example below shows the config creation:
Config file 'tesla-history.conf' not found

Do you want to create the config now? [Y/n] Y

Tesla Account Setup
-------------------
Email address<EMAIL_ADDRESS>Save auth token to: [tesla-history.auth]

InfluxDB Setup
--------------
Host: [localhost]
Port: [8086]
User (leave blank if not used): [blank]
Pass (leave blank if not used): [blank]
Database: [powerwall]
Timezone (e.g. America/Los_Angeles): Australia/Sydney

Config saved to 'tesla-history.conf'

In most cases, the [default] values will be correct and can be accepted by pressing Enter, however these can be changed if you have a custom setup.
Generally, only your Tesla account email address and your timezone will be required.
After the config is saved, you will be prompted to login to your Tesla account.
This is done by opening the displayed URL in your browser and then logging in:
----------------------------------------
Tesla account<EMAIL_ADDRESS>----------------------------------------
Open the below address in your browser to login.

<copy URL to browser> e.g.: https://auth.tesla.com/oauth2/v3/authorize?response_type=code...etc.

After login, paste the URL of the 'Page Not Found' webpage below.

Enter URL after login: <paste URL from browser> e.g.: https://auth.tesla.com/void/callback?code=...etc.

After you have logged in successfully, the browser will show a 'Page Not Found' webpage. Copy the URL of this page and paste it at the prompt.
Once logged in successfully, you will be shown details of the energy site(s) associated with your account:
----------------------------------------
Tesla account<EMAIL_ADDRESS>----------------------------------------
      Site ID:<PHONE_NUMBER>
    Site name: My Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2022-10-13 22:40:59+11:00
----------------------------------------

Once these steps are completed, you should not have to login again.
Basic script usage and examples
To import history data from Tesla cloud for a given start/end period, use the --start and --end options (date/time range is inclusive and in format YYYY-MM-DD hh:mm:ss):
# Get history data for start/end period
python3 tesla-history.py --start "2022-10-01 00:00:00" --end "2022-10-05 23:59:59"

The above example would retrieve history data for the first 5 full days of October.
You can run in test mode first which will not import any data, by using the --test option:
# Run in test mode with --test (will not import data)
python3 tesla-history.py --start "2022-10-01 00:00:00" --end "2022-10-05 23:59:59" --test

Example output:
----------------------------------------
Tesla account<EMAIL_ADDRESS>----------------------------------------
      Site ID:<PHONE_NUMBER>
    Site name: My Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2022-10-13 22:40:59+11:00
----------------------------------------
Running for period: [2022-10-01 00:00:00+10:00] - [2022-10-05 23:59:59+11:00] (4 days, 22:59:59s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2022-10-02 11:21:00+11:00] - [2022-10-02 12:41:00+11:00] (1:20:00s)
* Found data gap: [2022-10-04 06:09:00+11:00] - [2022-10-04 06:45:00+11:00] (0:36:00s)
* Found data gap: [2022-10-04 12:29:00+11:00] - [2022-10-04 14:56:00+11:00] (2:27:00s)

Searching InfluxDB for data gaps (grid status)
* Found data gap: [2022-10-02 11:21:00+11:00] - [2022-10-02 12:41:00+11:00] (1:20:00s)
* Found data gap: [2022-10-04 06:09:00+11:00] - [2022-10-04 06:45:00+11:00] (0:36:00s)
* Found data gap: [2022-10-04 12:29:00+11:00] - [2022-10-04 14:56:00+11:00] (2:27:00s)

Retrieving data for gap: [2022-10-02 11:22:00+11:00] - [2022-10-02 12:40:59+11:00] (1:18:59s)
* Loading daily history: [2022-10-02]
Retrieving data for gap: [2022-10-04 06:10:00+11:00] - [2022-10-04 06:44:59+11:00] (0:34:59s)
* Loading daily history: [2022-10-04]
Retrieving data for gap: [2022-10-04 12:30:00+11:00] - [2022-10-04 14:55:59+11:00] (2:25:59s)

Retrieving backup event history
* Creating grid status data: [2022-10-02 11:22:00+11:00] - [2022-10-02 12:40:59+11:00] (1:18:59s)
* Creating grid status data: [2022-10-04 06:10:00+11:00] - [2022-10-04 06:44:59+11:00] (0:34:59s)
* Creating grid status data: [2022-10-04 12:30:00+11:00] - [2022-10-04 14:55:59+11:00] (2:25:59s)

Writing to InfluxDB (*** skipped - test mode enabled ***)
Updating InfluxDB (*** skipped - test mode enabled ***)
Done.

If backup events are identified, this will be shown in the output, and the imported grid status data will include the outages:
----------------------------------------
Tesla account<EMAIL_ADDRESS>----------------------------------------
      Site ID:<PHONE_NUMBER>
    Site name: My Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2022-10-13 22:40:59+11:00
----------------------------------------
Running for period: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)

Searching InfluxDB for data gaps (grid status)
* Found data gap: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)

Retrieving data for gap: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)
* Loading daily history: [2022-04-03]
* Loading daily history: [2022-04-04]
* Loading daily history: [2022-04-05]
* Loading daily history: [2022-04-06]
* Loading daily history: [2022-04-07]
* Loading daily history: [2022-04-08]
* Loading daily history: [2022-04-09]
* Loading daily history: [2022-04-10]
* Loading daily history: [2022-04-11]
* Loading daily history: [2022-04-12]
* Loading daily history: [2022-04-13]
* Loading daily history: [2022-04-14]
* Loading daily history: [2022-04-15]
* Loading daily history: [2022-04-16]
* Loading daily history: [2022-04-17]
* Loading daily history: [2022-04-18]
* Loading daily history: [2022-04-19]
* Loading daily history: [2022-04-20]

Retrieving backup event history
* Creating grid status data: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)
* Found backup event period: [2022-04-19 20:55:53+10:00] - [2022-04-19 22:00:16+10:00] (1:04:23s)
* Found backup event period: [2022-04-19 20:53:39+10:00] - [2022-04-19 20:54:46+10:00] (0:01:07s)
* Found backup event period: [2022-04-08 19:00:14+10:00] - [2022-04-08 19:02:55+10:00] (0:02:41s)
* Found backup event period: [2022-04-08 18:57:32+10:00] - [2022-04-08 18:58:28+10:00] (0:00:56s)
* Found backup event period: [2022-04-08 18:54:56+10:00] - [2022-04-08 18:56:21+10:00] (0:01:25s)
* Found backup event period: [2022-04-04 21:16:45+10:00] - [2022-04-04 21:19:10+10:00] (0:02:25s)

Writing to InfluxDB
Updating InfluxDB
Done.

Since grid status logging was only added to Powerwall-Dashboard in June, you can use the tool to import grid status history from before this time, without affecting your existing power usage data.
The search for missing power usage / grid status data is done independently, so power usage history retrieval will be skipped and the missing grid status data will still be retrieved from Tesla cloud and imported:
----------------------------------------
Tesla account<EMAIL_ADDRESS>----------------------------------------
      Site ID:<PHONE_NUMBER>
    Site name: My Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2022-10-13 22:40:59+11:00
----------------------------------------
Running for period: [2022-04-03 00:00:00+11:00] - [2022-04-14 00:00:00+10:00] (11 days, 1:00:00s)

Searching InfluxDB for data gaps (power usage)
* None found

Searching InfluxDB for data gaps (grid status)
* Found data gap: [2022-04-03 00:00:00+11:00] - [2022-04-14 00:00:00+10:00] (11 days, 1:00:00s)

Retrieving backup event history
* Creating grid status data: [2022-04-03 00:00:00+11:00] - [2022-04-14 00:00:00+10:00] (11 days, 1:00:00s)
* Found backup event period: [2022-04-08 19:00:14+10:00] - [2022-04-08 19:02:55+10:00] (0:02:41s)
* Found backup event period: [2022-04-08 18:57:32+10:00] - [2022-04-08 18:58:28+10:00] (0:00:56s)
* Found backup event period: [2022-04-08 18:54:56+10:00] - [2022-04-08 18:56:21+10:00] (0:01:25s)
* Found backup event period: [2022-04-04 21:16:45+10:00] - [2022-04-04 21:19:10+10:00] (0:02:25s)

Writing to InfluxDB
Done.

Some convenience date options are also available (e.g. these could be used via cron):
# Convenience date options (both options can be used in a single command if desired)
python3 tesla-history.py --today
python3 tesla-history.py --yesterday

Finally, if something goes wrong, the imported data can be removed from InfluxDB with the --remove option. Data logged by Powerwall-Dashboard will not be affected, only imported data will be removed:
# Remove imported data
python3 tesla-history.py --start "YYYY-MM-DD hh:mm:ss" --end "YYYY-MM-DD hh:mm:ss" --remove

For more usage options, run without arguments or with the --help option:
# Show usage options
python3 tesla-history.py --help

usage: tesla-history.py [-h] [-l] [-t] [-d] [--config CONFIG] [--site SITE] [--ignoretz] [--force] [--remove] [--start START] [--end END] [--today] [--yesterday]

Import Powerwall history data from Tesla Owner API (Tesla cloud) into InfluxDB

options:
  -h, --help       show this help message and exit
  -l, --login      login to Tesla account only and save auth token (do not get history)
  -t, --test       enable test mode (do not import into InfluxDB)
  -d, --debug      enable debug output (print raw responses from Tesla cloud)

advanced options:
  --config CONFIG  specify an alternate config file (default: tesla-history.conf)
  --site SITE      site id (required for Tesla accounts with multiple energy sites)
  --ignoretz       ignore timezone difference between Tesla cloud and InfluxDB
  --force          force import for date/time range (skip search for data gaps)
  --remove         remove imported data from InfluxDB for date/time range

date/time range options:
  --start START    start date and time ("YYYY-MM-DD hh:mm:ss")
  --end END        end date and time ("YYYY-MM-DD hh:mm:ss")
  --today          set start/end range to "today"
  --yesterday      set start/end range to "yesterday"

Advanced option notes

--debug can be used to enable debug output. This will print the raw responses from Tesla cloud which might be helpful in some circumstances
--force option can be used to import data regardless of existing data (i.e. the search for data gaps is skipped). Should not be required normally, but could be useful for testing purposes
--remove will remove any previously imported data from InfluxDB for the date/time range, without affecting data logged by Powerwall-Dashboard
--ignoretz if your Tesla Powerwall timezone configuration is incorrect, this option may be required. Enabling it will ignore timezone differences between Tesla and InfluxDB. I'm not sure if this would ever happen without more real-world data on installs - could an installer have configured it wrong? The program will give an error if the detected Tesla site timezone doesn't match your configured InfluxDB timezone (please do check you have configured the InfluxDB timezone correctly if you get this error). If the Tesla timezone is in fact wrong (installer error?), I'd be interested to know. Using this option is untested, but I did attempt to account for it in the program by converting datetimes from the configured InfluxDB timezone to the detected Tesla site timezone and vice versa (it should work...)

InfluxDB notes and Analysis Data
I am recording some technical notes below regarding InfluxDB and the Analysis data updates for anyone interested - although mainly because I'm likely to forget why I have done something a certain way in the program if I need to look at this again in the future. :blush:
Importing data:

When writing large amounts of data to InfluxDB, I found it is significantly faster to write data points in Line Protocol format, instead of JSON
The power usage data retrieved from Tesla cloud is imported into autogen.http however with an added tag of source='cloud'
Any existing data points will not be affected at all, as the tags don't match (refer here for an explanation of how InfluxDB handles duplicate data points)
This also means, it is easy to remove any imported data by simply specifying WHERE source='cloud' in a DELETE query - only imported data will be removed, as original data will not have this tag
When deleting data in InfluxDB, it is important to note it is not possible to remove data from a specific retention policy, instead a DELETE will remove data from all retention policies. However, due to the imported data being tagged with source='cloud' this will not be an issue as the data in the other retention policies will not have this tag

For updating analysis data of InfluxDB (kwh, daily, and monthly retention policies):

After data is imported into autogen, the script will currently replicate the SELECT INTO queries used in the CQ's for kwh, daily, and monthly
To update the measurement values of existing data points however, it is important to ensure the data points have the exact same timestamp and tags (refer here again), therefore points cannot be tagged with source='cloud' as this would add points (NOTE: some of the CQ's are a sum() of data points, so adding additional points would result in e.g. doubling of values)
So as not to affect the analysis data unnecessarily, the tool will only execute the update queries for time ranges where new data points have been imported (+/- the group by time)
Finally just noting, there has been some discussion in #87 regarding the accuracy of the analysis data using integral calculations, so the script may need updating once a more accurate method has been determined

Some final notes for anyone using this tool to potentially import a lot of historical data:
If you have a lot of historical data you would like to import, it is probably advisable to split this up into batches of no more than few months at a time. During my testing, when retrieving the daily history data, I found the Tesla cloud site may stop responding if you overload it with too many requests in short periods (this is noted in the TeslaPy module Be aware that Tesla may temporarily block your account if you are hammering the servers too much).
However, this did only occur when trying to request data as quickly as possible. As such, a default 1 second delay between requests has been added to the script (can be changed by adding DELAY = x in the config file), after which I was able to retrieve a full year of historical data in one run without issue.
Please let us know if you find any issues or have questions.

This looks fantastic. I haven't had a chance to test it yet, but I want to soon. I love that you have it designed to search for and fill gaps. That's great.
Thanks for your work. I look forward to testing.

AMAZING!!! I imported over six months of data and filled a ton of small data gaps. No issues so far. THANKS!!!

Cool! I can't wait to try it out. I assume your work is based on the most current release schema, so I should upgrade first?

Cool! I can't wait to try it out. I assume your work is based on the most current release schema, so I should upgrade first?

@mikkyo9 - You will need to be running at least version 2.3.0 I believe, as that was the release when the CQ for grid status was added (released July). I'd still recommend upgrading to the latest version if you can though, as there has been many fixes.

AMAZING!!! I imported over six months of data and filled a ton of small data gaps. No issues so far. THANKS!!!

@jgleigh -
Awesome to hear, thanks for the feedback!
Yes, you may see a number of small data gaps identified even if you think your system had been logging non-stop without issues.
This is more likely with the grid status data as the gap threshold is 1 minute for that, as apposed to 5 minutes for power usage.
Any time you do an upgrade, for instance, you probably lose a few minutes of data but generally not more than 5 mins in my experience.
I considered accepting longer gaps for grid data - but then looking at my own backup history, I've had a number of events that range from a few seconds to a few minutes... so this way these will still be filled in if the system did not log data during that time for whatever reason.

Great job on this!
Worked very well filling dozens of gaps and a missing year of data.

Works like a charm. Thanks for putting this together @mcbirse

Worked quite well for me also, thanks for the all your efforts!

This worked like a charm. It's amazing what a community can do. I have all the date from system install August 2021 thanks @mcbirse and @jasonacox

Thanks everyone for the feedback - great to know it is working! 😊
I had one last TODO item for the program, which was to confirm the home power usage calculation is correct during VPP events (it should be 🤞 ).
@cwagz - as discussed earlier, you have some data from a VPP event which should help us to confirm this.
If possible, it would be great if you could run the program with the below arguments when you have a chance:
python3 tesla-history.py --start "2022-08-17 00:00:00" --end "2022-08-17 23:59:59" --force --debug --test

Using the --test option will ensure no data is imported so it won't affect your system.
--debug will show the responses from Tesla cloud. I am interested in seeing the power usage responses only, like below:
(i.e. not the other responses, which could include personal information about your powerwall)
        {
            "timestamp": "2022-08-17T18:00:00+10:00",
            "solar_power": 0,
            "battery_power":<PHONE_NUMBER>428571,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },
        {
            "timestamp": "2022-08-17T18:05:00+10:00",
            "solar_power": 0,
            "battery_power":<PHONE_NUMBER>142857,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },
        {
            "timestamp": "2022-08-17T18:10:00+10:00",
            "solar_power": 0,
            "battery_power":<PHONE_NUMBER>333334,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },

Specifically I would like to see the values around 6pm-8pm (and before/after) when the VPP event occurred, so we can see in the response how the grid_services_power value is used and whether that affects how home usage power is calculated:

Appreciate the help, thanks.

@mcbirse - Here you go. Thanks for working on this!
    {
        "timestamp": "2022-08-17T17:55:00-07:00",
        "solar_power": 1727.142857142857,
        "battery_power": 0,
        "grid_power": 1146,
        "grid_services_power": 0,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T18:00:00-07:00",
        "solar_power": 1592.857142857143,
        "battery_power": 11457.142857142857,
        "grid_power": -6196.857142857143,
        "grid_services_power": -10547.370702489794,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T18:05:00-07:00",
        "solar_power": 1501.6666666666667,
        "battery_power": 11460,
        "grid_power": -7958.333333333333,
        "grid_services_power": -11602.287772738773,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T18:10:00-07:00",
        "solar_power": 1405.7142857142858,
        "battery_power": 11454.285714285714,
        "grid_power": -10301.714285714286,
        "grid_services_power": -11602.387772738773,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T18:15:00-07:00",
        "solar_power": 1308.5714285714287,
        "battery_power": 11455.714285714286,
        "grid_power": -10424.285714285714,
        "grid_services_power": -11602.337772738774,
        "generator_power": 0
    },
	{
        "timestamp": "2022-08-17T19:30:00-07:00",
        "solar_power": 0,
        "battery_power": 11460.140845070422,
        "grid_power": -3353.760563380282,
        "grid_services_power": -10665.09230304092,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:35:00-07:00",
        "solar_power": 0,
        "battery_power": 11461.346153846154,
        "grid_power": -2332.6923076923076,
        "grid_services_power": -11600.851106072108,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:40:00-07:00",
        "solar_power": 0,
        "battery_power": 11455.757575757576,
        "grid_power": -7867.212121212121,
        "grid_services_power": -11600.841106072108,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:45:00-07:00",
        "solar_power": 0,
        "battery_power": 10071.919191919193,
        "grid_power": -7377.080808080808,
        "grid_services_power": -10440.758995464897,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:50:00-07:00",
        "solar_power": 0,
        "battery_power": 0,
        "grid_power": 2383.746835443038,
        "grid_services_power": 0,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:55:00-07:00",
        "solar_power": 0,
        "battery_power": 0,
        "grid_power": 6388.857142857143,
        "grid_services_power": 0,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T20:00:00-07:00",
        "solar_power": 0,
        "battery_power": 0,
        "grid_power": 6504.666666666667,
        "grid_services_power": 0,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T20:05:00-07:00",
        "solar_power": 0,
        "battery_power": 0,
        "grid_power": 3018.285714285714,
        "grid_services_power": 0,
        "generator_power": 0

@mcbirse - I just loaded all my history. This is amazing! Thank you again for doing this.

Does anyone else have some crazy data in the Tesla cloud? This is mine:

The sun must have really been blasting!
Would this be easy to clean up manually?

The sun must have really been blasting!

It's hard to read your tone here - I take it that you are having a chuckle about unexpected and unlikely milliwatt hours values in the data (and that was probably not a heavy cloud/rain day with minimal generation, also not Megawatt hours?)
You can use TeslaPy to take a look historical data - used by mcbirse in his history tool, and I'm having a putter with it on the historical cumulatives in the cloud.

Actually, I mistakenly thought this was the day I produced Megawatt hours. I should have checked the app better and done a sanity check on my math. The weird data day was actually December 14, 2021. The Tesla App shows .136 MWh that day. It seems my memory was off by an order of magnitude.
Tesla App:

Downloaded csv From Tesla App:

Interestingly, the anomaly did not get picked up when I imported the pre-Powerwall Dashboard data from the Tesla cloud:
Imported Data:

@cwagz - thanks for the output!
That's terrific, I believe this confirms all the calcs tally up with your graph correctly for home usage during the VPP.
So home usage power calculation is correct as below, no change required in the program (except to remove the comment):
# Calculate power usage values (TODO: check 'home' power usage calculation is correct during VPP event - NOTE: response includes 'grid_services_power')
home = d['solar_power'] + d['battery_power'] + d['grid_power']

E.g. at 6pm
    {
        "timestamp": "2022-08-17T18:00:00-07:00",
        "solar_power": 1592.857142857143,
        "battery_power": 11457.142857142857,
        "grid_power": -6196.857142857143,
        "grid_services_power": -10547.370702489794,
        "generator_power": 0,
    },

So at 6pm home usage in being calculated in the program as below:
"home" = "solar_power": 1592.86 + battery_power": 11457.14 + "grid_power": -6196.86 = 6853.14

This looks about right according to the graph at that time. Cheers!
P.S. @cwagz - Mass Effect fan? 😊

Getting this error.
ERROR: Failed to retrieve SITE_DATA - string indices must be integers

The Tesla App shows .136 MWh that day. It seems my memory was off by an order of magnitude.

The app and Tesla cloud do some odd things - you can get slightly different outcomes depending on the time range and the data you are pulling, and there are also a set of rules for "correcting" weird data that are not applied consistently (e.g. I have a day with a misconfigured CT - it can be reported with truncated data, with 'fixes" applied, or not reported at all, depending on the format you ask for).
I believe the history that @mcbirse extracts is reconstructed from 5 minute power readings, which I think are be a different data set to that used for the app energy reporting (as noted above - these can contain some weirdness). Lots of guesswork in closed APIs though!

Getting this error.
ERROR: Failed to retrieve SITE_DATA - string indices must be integers

@TechCrazi - strange, not sure why you would be getting this error (first error reported!).
Can you do some tests to help diagnose this issue?
First off, I'm just wondering if this was a temporary problem and it has subsequently started working at a later time, or is this happening all the time?
The script is failing on this API call:
# Retrieve site current time
data = battery.api('SITE_DATA')['response']

The response returned is a string instead of the expected JSON response. The string may contain some clues as to why it is failing, however I did not account for this so we can't see the message.
Can you test the script with the below command, and confirm you see some JSON output with your site details etc.
# Login to Tesla cloud and print debug output
python3 tesla-history.py --login --debug

If it is still failing with an error after the site details, perhaps try deleting tesla-history.auth and then re-running the script with --login to re-create the auth token, and see if that resolves it?
If not, there are some changes I can make to the script which might help, as getting the current site time is not critical to it's operation. I originally did this to confirm the system time of my Powerwall was valid and correct (I'm quite paranoid about date/times and ensuring they are right!!).
Just speculating...(feedback welcome) the API call for SITE_DATA is requesting the "live status" of the system, as below:
  "SITE_DATA": {
    "TYPE": "GET",
    "URI": "api/1/energy_sites/{site_id}/live_status",
    "AUTH": true
  },

Therefore I wonder: if a site had not reported "live status" for a period (e.g. loss of connectivity) this API call may return an error string?
It's basically guesswork with the Tesla cloud API unfortunately, since nothing is documented. I'll make some changes to work around this potential issue regardless, since it appears to have happened at least once, as in this case.

Getting this error.
ERROR: Failed to retrieve SITE_DATA - string indices must be integers

@TechCrazi - strange, not sure why you would be getting this error (first error reported!).
Can you do some tests to help diagnose this issue?
First off, I'm just wondering if this was a temporary problem and it has subsequently started working at a later time, or is this happening all the time?
The script is failing on this API call:
# Retrieve site current time
data = battery.api('SITE_DATA')['response']

The response returned is a string instead of the expected JSON response. The string may contain some clues as to why it is failing, however I did not account for this so we can't see the message.
Can you test the script with the below command, and confirm you see some JSON output with your site details etc.
# Login to Tesla cloud and print debug output
python3 tesla-history.py --login --debug

If it is still failing with an error after the site details, perhaps try deleting tesla-history.auth and then re-running the script with --login to re-create the auth token, and see if that resolves it?
If not, there are some changes I can make to the script which might help, as getting the current site time is not critical to it's operation. I originally did this to confirm the system time of my Powerwall was valid and correct (I'm quite paranoid about date/times and ensuring they are right!!).
Just speculating...(feedback welcome) the API call for SITE_DATA is requesting the "live status" of the system, as below:
  "SITE_DATA": {
    "TYPE": "GET",
    "URI": "api/1/energy_sites/{site_id}/live_status",
    "AUTH": true
  },

Therefore I wonder: if a site had not reported "live status" for a period (e.g. loss of connectivity) this API call may return an error string?
It's basically guesswork with the Tesla cloud API unfortunately, since nothing is documented. I'll make some changes to work around this potential issue regardless, since it appears to have happened at least once, as in this case.

@mcbirse Here is the debug output of my login. I have removed some of my personal info for privacy. I have 2 sites under my account. One of them was the wrong one that Tesla set it up on my account but I can not figure out how to delete it. I also tested the "--site" parameter but the site ID that is getting logged in debug is not the right one and I am not sure how to get the right site ID.
root@PowerWall:/Powerwall-Dashboard/tools# python3 tesla-history.py -l -d
Tesla account<EMAIL_ADDRESS>{
"id": "1538000-25-F--TG121182002WXC",
"site_name": "Ghar",
"backup_reserve_percent": 35,
"default_real_mode": "self_consumption",
"installation_date": "2021-08-11T19:00:00-05:00",
"user_settings": {
"storm_mode_enabled": true,
"powerwall_onboarding_settings_set": true,
"sync_grid_alert_enabled": true,
"breaker_alert_enabled": false
},
"components": {
"solar": true,
"solar_type": "pv_panel",
"battery": true,
"grid": true,
"backup": true,
"gateway": "teg",
"load_meter": true,
"tou_capable": true,
"storm_mode_capable": true,
"flex_energy_request_capable": false,
"car_charging_data_supported": false,
"off_grid_vehicle_charging_reserve_supported": true,
"vehicle_charging_performance_view_enabled": false,
"vehicle_charging_solar_offset_view_enabled": false,
"battery_solar_offset_view_enabled": true,
"solar_value_enabled": true,
"energy_value_header": "Energy Value",
"energy_value_subheader": "Estimated Value",
"energy_service_self_scheduling_enabled": true,
"show_grid_import_battery_source_cards": true,
"set_islanding_mode_enabled": true,
"wifi_commissioning_enabled": true,
"backup_time_remaining_enabled": true,
"rate_plan_manager_supported": true,
"battery_type": "solar_powerwall",
"configurable": true,
"grid_services_enabled": false,
"disallow_charge_from_grid_with_solar_installed": true,
"ITC_cliff": 1,
"edit_setting_permission_to_export": true,
"edit_setting_grid_charging": true,
"edit_setting_energy_exports": true
},
"version": "22.26.1 1da541cb",
"battery_count": 3,
"tariff_content": {
"code": "Tesla_TOU(edited)",
--- personal info removed ---
{
"id": "1538000-25-F--TG1211410020TG",
"site_name": "zzzDeleteMePlease-FakeSystem",
"backup_reserve_percent": 100,
"default_real_mode": "self_consumption",
"installation_date": "2021-07-16T18:30:59-05:00",
"user_settings": {
"storm_mode_enabled": true,
"powerwall_onboarding_settings_set": true,
"sync_grid_alert_enabled": false,
"breaker_alert_enabled": false
},
"components": {
"solar": true,
"battery": true,
"grid": true,
"backup": true,
"gateway": "teg",
"load_meter": true,
"tou_capable": true,
"storm_mode_capable": true,
"flex_energy_request_capable": false,
"car_charging_data_supported": false,
"off_grid_vehicle_charging_reserve_supported": false,
"vehicle_charging_performance_view_enabled": false,
"vehicle_charging_solar_offset_view_enabled": false,
"battery_solar_offset_view_enabled": true,
"solar_value_enabled": true,
"energy_value_header": "Energy Value",
"energy_value_subheader": "Estimated Value",
"energy_service_self_scheduling_enabled": true,
"show_grid_import_battery_source_cards": true,
"backup_time_remaining_enabled": true,
"rate_plan_manager_supported": true,
"battery_type": "solar_powerwall",
"configurable": true,
"grid_services_enabled": false,
"ITC_cliff": 1
},
"battery_count": 2,
"installation_time_zone": "America/Chicago",
"max_site_meter_power_ac":<PHONE_NUMBER>,
"min_site_meter_power_ac": -1000000000,
"geolocation": {
"latitude": 32.9701219,
"longitude": -96.5766098
},

@TechCrazi - Interesting!
Thanks, I can resolve this issue and should have an update to the program this weekend.
I suspect it is failing due to the additional (incorrect) site on your account. The "live_status" data retrieval to get current system time is probably failing, as the invalid site is not live / reporting data.
No problem - I'll make some changes which should allow it to report the sites without failure, and you will probably need to use the --site option to select the correct site (once fixed, it will output the site id correctly that you will need to use).

@mcbirse I called Tesla and they fixed my issue with duplicate sites. Now the script is working as expected...

@mcbirse I called Tesla and they fixed my issue with duplicate sites. Now the script is working as expected...

No worries @TechCrazi - good to hear it was fixed and thanks for the update!
I'll make a couple of small changes to the script regardless to try to work around this potential problem for any future users. If it requires contacting Tesla or an agent to get an invalid site removed, that could be a pain for some people.

New home (URL) for this great tool:

GitHub: https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/tesla-history
GH Pages: https://jasonacox.github.io/Powerwall-Dashboard/tools/tesla-history/

Just a heads up I had issues running the python3 tesla-history.py --login script on Windows using Git Bash. After I pasted the callback URL after login it would just get stuck/not do anything.
I used Windows command prompt instead and worked correctly. In case anyone else is trying to run this on Windows.

It seems like I have a period of time where my data is weird. Specifically, 2/25/22 to 2/28/22. This may have been a time when I had backed up and then reinstalled my dashboard. I am not really sure how to fix it, but I thought it would be appropriate to ask here. Basically, the usage / production that is logged for the day does not match the "realtime" usage. This created this couple of days where it looks like I was offline on the 12-month chart:

You can see on 2/26 a good example of the two measurements being off. I checked a range of time and there seems to be a problem with my data from 2/25 to 2/28. Both 2/24 and 3/1 match within reason.

How can I fix this in my data?

@cwagz - I think I follow the issue you are seeing. Please correct me if I have misunderstood.
Your "realtime" data (as shown in the Energy Usage panel) appears to be correct, however the corresponding cumulative data that should be shown for those days in Home Usage, Solar Energy, and Solar Energy Year is NOT correct?
The data for the "Monthly Analysis" panels is queried from the kwh retention policy. The kwh data is extrapolated from the your "realtime" data, i.e. from autogen.
So if your data in autogen is all okay, you can run the queries that populate kwh, daily, and monthly, manually for that period (these are normally run automatically from the CQ definitions - continuous queries). I wonder how this issue has occurred?
Try the following to correct the data:
# Run the InfluxDB shell console
docker exec -it influxdb influx -database 'powerwall'

This allows you to connect to InfluxDB and manually run queries. You should see the below:
Connected to http://localhost:8086 version 1.8.10
InfluxDB shell version: 1.8.10
>

At the > prompt, then run the following queries (they need to be run in the order shown below - that is into kwh, then daily, then monthly).
Note - you need to edit the below queries for your timezone!!
Please replace the timezone strings with your own. And replace the timezone offset in the WHERE time clauses with your own offset for those dates (you may need to check what it was on those dates due to DST etc. For Los Angeles, I guess it was -8 on those dates, I think).
Paste queries at the > prompt, and press Enter.
SELECT from autogen into kwh:
SELECT integral(home)/1000/3600 AS home, integral(solar)/1000/3600 AS solar, integral(from_pw)/1000/3600 AS from_pw, integral(to_pw)/1000/3600 AS to_pw, integral(from_grid)/1000/3600 AS from_grid, integral(to_grid)/1000/3600 AS to_grid INTO kwh.:MEASUREMENT FROM autogen.http WHERE time >= '2022-02-25T00:00:00-08:00' AND time < '2022-03-01T00:00:00-08:00' GROUP BY time(1h), month, year tz('America/Los_Angeles')

SELECT from kwh into daily:
SELECT sum(home) AS home, sum(solar) AS solar, sum(from_pw) AS from_pw, sum(to_pw) AS to_pw, sum(from_grid) AS from_grid, sum(to_grid) AS to_grid INTO daily.:MEASUREMENT FROM kwh.http WHERE time >= '2022-02-25T00:00:00-08:00' AND time < '2022-03-01T00:00:00-08:00' GROUP BY time(1d), month, year tz('America/Los_Angeles')

SELECT from daily into monthly:
SELECT sum(home) AS home, sum(solar) AS solar, sum(from_pw) AS from_pw, sum(to_pw) AS to_pw, sum(from_grid) AS from_grid, sum(to_grid) AS to_grid INTO monthly.:MEASUREMENT FROM daily.http WHERE time >= '2022-02-01T00:00:00-08:00' AND time < '2022-03-01T00:00:00-08:00' GROUP BY time(365d), month, year

After running each query you should get a response like below. The "written" value will be different for each due to the different group by times.
Response example:
name: result
time written
---- -------
0    96

Hopefully, after doing this, when you check the graphs in Grafana the Analysis Data values will be corrected? 🤞

@mcbirse - Thank you for helping with this! You understood the problem perfectly. I ran your commands as written. The time zone and offset were correct for that time of year. The days now look good in Grafana.
I do not recall when I first started using the dashboard, but it was fairly close to where this bad data existed. I changed hardware several times in the beginning until I ended up with a setup I like that is also fast enough. I am guessing I had an incomplete backup or something from that time period that got moved onto new hardware. I really don't know though, this is my first experience with InfluxDB and I need to read up on how it works.
I was using a PC Engines APU2C4 for a few weeks, but it would struggle really bad if I tried to pull up multiple days' worth of data in Grafana. Would something like the problem I had occur with InfluxDB if the CPU was maxed out?

@cwagz - Great to hear this fixed the issue!

Would something like the problem I had occur with InfluxDB if the CPU was maxed out?

I suppose it is possible, but I'm not sure. It's a strange issue to have occurred, since the real time data was still logging okay.
There is a difference with the CQ's for kwh, daily, monthly, in that compared to other CQ's they use a "RESAMPLE EVERY 1m" clause - maybe this resampling failed for some reason, like the high CPU load?
Actually, I just Googled for that and found reference to this possibility. If the query is taking longer than the resample time (1m in this case) then it will fail silently and the data in kwh etc. would not have been updated.
You mentioned it struggled to pull data into Grafana on that system, so it sounds like queries were taking a long time. So it seems highly likely the resample queries were failing and caused this issue.

@cwagz or for anyone else's interest:
There is a way to check if CQ's are failing by querying InfluxDB's internal database. This could be useful for anyone setting up a new host and wanting to check if it is handling the load okay?
Unfortunately it appears to only store a week of history - so @cwagz we cannot confirm for sure what caused your issue.
# Connect to InfluxDB internal database
docker exec -it influxdb influx -database '_internal' -precision 'rfc3339'

Query first 5 cq records:
SELECT * FROM cq LIMIT 5 tz('Australia/Sydney')

Response showing queryFail and queryOk counts:
name: cq
time                      hostname queryFail queryOk
----                      -------- --------- -------
2022-10-23T11:00:00+11:00 influxdb 0         139012
2022-10-23T11:00:10+11:00 influxdb 0         139031
2022-10-23T11:00:20+11:00 influxdb 0         139031
2022-10-23T11:00:30+11:00 influxdb 0         139031
2022-10-23T11:00:40+11:00 influxdb 0         139031

Query last 5 cq records (most recent):
SELECT * FROM cq ORDER BY DESC LIMIT 5 tz('Australia/Sydney')

Response:
name: cq
time                      hostname queryFail queryOk
----                      -------- --------- -------
2022-10-31T11:02:30+11:00 influxdb 0         192
2022-10-31T11:02:20+11:00 influxdb 0         192
2022-10-31T11:02:10+11:00 influxdb 0         192
2022-10-31T11:02:00+11:00 influxdb 0         175
2022-10-31T11:01:50+11:00 influxdb 0         175

Maybe I should open another discussion thread at some point where we can document various InfluxDB tips & tricks.

Maybe I should open another discussion thread at some point where we can document various InfluxDB tips & tricks.

This is probably a good idea. All of this information is very helpful. Thanks again.

@mcbirse thank you so much for the script and instructions– I ran it to backfill the entire past year and everything came through exactly as intended first try. I've wanted to run this dashboard for a while– and figured I would set it up for 2023. The fact I was just able to backfill everything with a single script is amazing. 🙇 🙌
Thank you too for the dashboard @jasonacox, this is very neat! 🙇

@chrisheninger - Thanks for the feedback, awesome to hear! Happy New Year! 🥳 🎉

Hi Like @TechCrazi, i have registered on my account 2 powerwalls, one is an old system still registered in the account, (offline for months now), when listing the sites, the old one is listed correctly, when the script need to read the second powerwall info (the one that i currently own and that is running) i get this error:
Get SITE_DATA for Site ID 2352913 {     "response": "" } ERROR: Failed to retrieve SITE_DATA - string indices must be integers
someone have some idea on what to try?

Hi @jurassic410 - I have not seen this occur before, but it appears the request for SITE_DATA is returning an empty response, which is a bit odd.
This hopefully shouldn't be too difficult to account for, and that request is only used to get the current system time of your Powerwall (which, when not available, should fall back to the current system time of your host server instead).
I'll try to submit a PR for the program this weekend that should fix this issue.

@mcbirse thank you, 8 will look forward to it, and let you know if the fix works!

@jurassic410 - I have submitted a fix for this (hopefully). Once @jasonacox has merged the changes and you have pulled the update, let me know how it goes! 🤞
Essentially the request for SITE_DATA has been changed, so if any exception at all occurs with this request the script will no longer exit.
The request was used simply to get the system time from the Powerwall, so is not very critical (and when unable to get this, the script would use the host system current time instead anyway).

Changes merged!  Thanks for the fix here and in the pypowerwall set-mode.py and set-reserve.py tools!

@mcbirse @jasonacox now works like a charm!
it will now asks correctly wich site to use.
thank you

Hi @Jot18 - for solar only sites, this is a work in progress at the moment. There is a modified version of the dashboard under the "tools/solar-only" folder here as well as modified tesla-history script that will work with solar sites.
Please give it a try - I would welcome some more feedback! Check/join the discussion #183 and let us know if it works for you. 🙏

Hi @Jot18 - for solar only sites, this is a work in progress at the moment. There is a modified version of the dashboard under the "tools/solar-only" folder here as well as modified tesla-history script that will work with solar sites.
Please give it a try - I would welcome some more feedback! Check/join the discussion #183 and let us know if it works for you. 🙏

Tried that as well. Couldn't really understand how to use that one. I only had success getting data with c[https://auth.tesla.com/oauth2/v3/authorize?response_type=code&client_id=ownerapi&redirect_uri=https%3A%2F%2Fauth.tesla.com%2Fvoid%2Fcallback&scope=openid+email+offline_access&state=P5hztmHgtWMFMwv7mObCdIivp7ASFx&code_challenge=LJjPtbFTRO_tsq7ZP7yxJKMtORnP6IPA-sqBNMmwvm0&code_challenge_method=S256&login_hint=Gurmailsaini12%40yahoo.com](url) but that also couldn't download data for dec 2022, Jan/Feb 2023. It only downloads data for Oct/Nov 2022 & March 2023. I got solar in late oct 2023.

@Jot18 - the script in the solar-only tools folder works pretty much the same. Make sure to read the usage instructions for the script.
Can you post some output/details of what you are running, errors received, etc.?
Try starting with below to ensure your config is set up correctly and the script finds your Tesla energy site correctly.
# Run tesla-history login
python3 tesla-history.py --login

Also, you may need run setup.sh from the "tools/solar-only" directory to set up Powerwall-Dashboard for a solar only site first (rather than the main project). Please post in #183 for further help with the solar-only testing.

@Jot18 - the script in the solar-only tools folder works pretty much the same. Make sure to read the usage instructions for the script.
Can you post some output/details of what you are running, errors received, etc.?
Try starting with below to ensure your config is set up correctly and the script finds your Tesla energy site correctly.
# Run tesla-history login
python3 tesla-history.py --login

Also, you may need run setup.sh from the "tools/solar-only" directory to set up Powerwall-Dashboard for a solar only site first (rather than the main project). Please post in #183 for further help with the solar-only testing.

After pasting the URL, I get no tesla energy sites found. I did run from Solar-Only>Tesla-Hitory.

THANK YOU to @mcbirse and the community so much for pulling this historical tool together.  I got PW Dashboard up and running a week ago, and over this past weekend I successfully fought through pulling in all my history since 2019.
I’m writing to share the extra step I had to do given that I’m using a Synology DS1618+ to host my Docker instances.  Just like with PW Dashboard itself, there is some funkiness to using the Tesla history app.  Hopefully I can save someone the trouble I went through to pull my old data.
As a recap for other aspiring Synology hosters, definitely make sure you follow this procedure posted by the Community (Docker issue on Synology NAS · jasonacox/Powerwall-Dashboard · Discussion #169 · GitHub).  In particular, make sure you:
sudo synogroup --add docker
sudo chown root:docker /var/run/docker.sock
sudo synogroup --member docker <username>

And finally it is vital that you hand modify the Grafana section of the file powerwall.yml that spins up all the Docker instances.  If you do not do this step, what happens is that the Grafana container gets stuck in a perpetual restart loop.

[ ] Edit “user 1000:1000” to change the default value of 1000 to whatever ID the username is
[ ] For example, in my case it became “user 1026:1026”

After that it’s a breeze and the whole package comes up nicely and starts populating the Grafana dashboard.  So…. now on to pulling historical data from Tesla.
I can confirm that the current iteration of the tesla-history.py tool runs well, thanks again to @mcbirse for doing this.  However, the Synology Python install is abnormal in several respects that breaks the procedure:

Synology bundles an old Python 2 with the OS, and then upgrades it automatically in the background with DSM updates.  It cannot be de-installed and you don’t want to modify it.
This Python instance does not include PIP, which is required for the tesla-history.py tool.

Note I am running latest DSM (DSM 7.1.1-42962 Update 4) which includes Python 3.8.12.
I attempted and had to roll back the following workarounds to get PIP running after they failed:

Install PIP as root (via both sudo -i and direct login as root)

This successfully installs PIP but due to Synology implementation will not run as anything other than root.
Exiting sudo and checking for PIP results in this error:

Since I don’t run anybody’s code as root that doesn’t require it, this was a dead end for me.  Had to back out PIP completely with “sudo pip uninstall pip”
NOTE – Synology specifically disclaims and warns again doing anything in Python as root and throws lots of warnings about it.  You’ll see this error every time:

Repeat above with various PIP installation techniques

Attempted “python3 -m ensurepip --default-pip”.  Failed and rolled it back.
Attempted “wget https://bootstrap.pypa.io/get-pip.py” followed by invoking “python3 get-pip.py”.   Failed and rolled it back.
When I say it failed, I mean that after leaving root and returning to my normal user CLI any attempt to invoke PIP failed with some version of the traceback error above.

Install Python 3.9 separately and then manually install PIP in that instance.

Python 3.9 is available through the Synology Package Center for automatic download & install.  This works fine, and when it’s done you can do invoke either “python3” or “python3.9” from CLI depending on which instance you want to hit.
However, this package also does not include PIP.  Attempting to install it as a non-root user always failed.  Attempting as root repeated the experience above with built-in Python bundle.
Since Py3.9 is a fully optional bundle, I attempted to see if I could play around with the directly structure directly and then ran into a variety of other problems.  The two biggest were that the DSM Package Center installs everything as root only (everything in site-packages directory is 700).  And python3.9 is not added to the user $PATH and it seems to require an act of god to modify this.
So I backed out Py3.9 altogether since I don’t need it for anything else and the Synology folks periodically update their version so want to avoid conflicts.

  The reasons I didn’t do this out of the gate are (1) it seemed like having a working PIP would be useful for other things; (2) I haven’t any experience with Python VENVs yet; and finally (3) I did not appreciate that the tesla_history.py process is a one-and-done.
So what ultimately worked just fine on my DS1618+ to pull my history was the following:
sudo -i
python3 -m venv <Virtual_Env_Name>
cd <Virtual_Env_Name>
source bin/activate
python3 -m pip install --upgrade pip
pip install python-dateutil teslapy influxdb

And from there you can run the entire tesla-history login and download procedure as laid out.  Note the whole thing is running inside the Python venv, but it doesn’t  matter since the way the script works is that it’s essentially a temporary buffer for data pulled via HTTPS.  Once the download is done for the specified time interval, it writes out everything it has to InfluxDB so there is no residual state required.  You can blow away the venv after you verify the history is showing up in Grafana.
I now have history all the way back to April of 2019 when my PW got installed, and I’m geeking out studying trends in the data.

Thanks again!

GREAT writeup! Thanks @clukas1967 !

sudo synogroup --add docker
sudo chown root:docker /var/run/docker.sock
sudo synogroup --member docker

I wonder if we should update the README.md instructions for Dashboard installation on Synology systems (here).  Would you want to submit a PR for that so you can get credit?
One change:

modify the Grafana section of the file powerwall.yml

That will break the upgrade.sh process.  Instead, you should be able to edit compose.env to do the same thing:
GRAFANAUSER="1000:1000"

Thanks!! 🙏

Thanks @jasonacox.  I should have also acknowledged you in my original post as we wouldn't have any of this without you!
I agree it makes sense to make the readme more explicit for PW Dashboard install.  But I don't deserve any credit since I merely restated the procedure in the link I cited in more compact form.  Regarding editing compose.env vs. powerwall.yml I would defer to your expertise here.
I do think it makes sense to have a Synology section in a readme specifically for the Tesla History utility.  You could just take the last CLI block from my post and append the admonition to follow it up with the original procedure offered by @mcbirse on 10/13/2022.  Probably that whole lengthy post should be the readme.
In a few days I will be starting a couple of new threads in here as I'm about to expand both my battery storage and my PV generation on the roof and have questions/suggestions about how that will work.

Thanks @clukas1967 - great to hear the script has been so useful!
Next iteration will support solar-only sites (as per version under tools/solar-only). This is already working now for those solar-only users and at some point I will merge all these changes to the primary script so we are not maintaining multiple versions... (still some QoL updates to do for solar-only users, such as a daemon/background mode).
Excellent info for the Synology NAS users out there, of which there appears to be many! Personally I'm running Powerwall-Dashboard in an Alpine Linux VM on a QNAP NAS (since I had that set up for other purposes already). The versatility of Powerwall-Dashboard and number of platforms it can be run on is fantastic.

Once the download is done for the specified time interval, it writes out everything it has to InfluxDB so there is no residual state required. You can blow away the venv after you verify the history is showing up in Grafana.

You could - but I find it useful to have the script on hand so it can be easily run at any time. I have frequently used it myself for instances where my host server was down (e.g. due to OS updates, NAS firmware updates, etc.), so you can then fill in missing data from the Tesla cloud at least, whenever you need to.

Just a heads up I had issues running the python3 tesla-history.py --login script on Windows using Git Bash. After I pasted the callback URL after login it would just get stuck/not do anything.
I used Windows command prompt instead and worked correctly. In case anyone else is trying to run this on Windows.

THANKS! Just had the same issue and this worked.

Not sure if it's due to a change on Tesla's end but haven't been able to get past the Tesla Account setup steps in the instructions. After copying the URL, loging into the Tesla Account and getting "Page Not Found" the URL that I get results in the following error: Login failure - (mismatching_state) CSRF Warning! State not equal in request and response.
Is there another way to add the authentication token to the script?

Hi @wiltocking  - A few things:

Try clearing your cookies in your browser and re-authenticating
Do you get any more information when you supply the --debug option to tesla-history.py?

Hi @wiltocking - A few things:

Try clearing your cookies in your browser and re-authenticating
Do you get any more information when you supply the --debug option to tesla-history.py?

Well, new day new results, cleared cookies again and this time the URL worked! Downloading Tesla cloud history now. Thank you :)

Awesome!  Thanks @wiltocking !

Im not able to run the script getting the error ERROR: Login failure - MissingTokenError('(missing_token) Missing access token parameter.') after login

I should also mention that my dashboard is up and running

Hi @kylebrowning  - I recommend following these steps: https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/tesla-history#setup-and-logging-in-to-tesla-account

I thought I'd try this; I'm away from home at the mo so doing it remotely and I get this error when I try to login. I have deleted the config file an started from the beginning and get the same error
ERROR: Failed to retrieve PRODUCT_LIST - LoginRequired('(login_required) Login required')

Interesting. Did you get any error from Tesla when you tried the login, or just the expected 404 page URL?

Hi All,
First, huge thanks to @jasonacox for the awesome dashboard, @mcbirse for the super useful history tool and @clukas1967 for the life saving tips on getting everything running on a Synology NAS!
Now the issue: I am in Sydney/Australia, my local system timezone is Sydney/Australia, as is my InfluxDB and Site timezone, but for some reason that I cannot work out, when I run tesla-history, datetime values are output as Sydney/Australia time values, but with UTC offset (ie. +00:00) instead of Sydney/Australia offset (+11:00 or +10:00, depending on Daylight savings).  As a consequence, the calls to get_calendar_history_data only return 10 or 11 hours of data and I end up with 13 or 14 hour gaps every day.
I have worked around the issue with a filthy hack (I changed day.replace(tzinfo=sitetz).isoformat() to f"{day.isoformat()}+11:00", noting the only gaps I had were during daylight savings), but I would like to work out what the issue is and fix it so that I can use the tool without the hack.
Any ideas?

Hi @porlcass - I take it you are running the tesla-history script on a Synology NAS?
There seems to be some issue with the python dateutil module obtaining the timezone correctly on Synology NAS systems.
Refer issue #455 and the last post here.
I'm not sure of the exact cause, but it's possible the timezone files on Synology NAS are messed up in some way or incompatible with python dateutil.
If you are on Synology (or even another system), it would be great if you could run some of the python tests mentioned to see where the issue is occurring.
A subset of those, like below, is probably sufficient in fact. Run python and paste all of the below commands in the python shell, and post the results.
from dateutil.parser import isoparse
from dateutil import tz

start = isoparse("2023-06-26 00:00:00")
start
print(start)
type(start)
type(start.utcoffset())

influxtz = tz.gettz("Australia/Sydney")
influxtz
print(influxtz)
type(influxtz)

start = start.replace(tzinfo=influxtz)
start
print(start)
type(start)
type(start.utcoffset())

It would be interesting to see what is returned for the final print(start), like the below.
>>> start = start.replace(tzinfo=influxtz)
>>> start
datetime.datetime(2023, 6, 26, 0, 0, tzinfo=tzfile('/usr/share/zoneinfo/Australia/Sydney'))
>>> print(start)
2023-06-26 00:00:00+00:00

The above (from a Synology NAS) is showing the timestamp is output with a +00:00 offset (UTC), which is incorrect based on the timezone specified. It should show +10:00 for Australia/Sydney for the above example.
Is your system showing a similar issue?
Can any other Synology NAS users shed some light on this issue? Unfortunately, I do not have a Synology NAS to test with myself.
The script is currently using the python dateutil module, however from python 3.9 the built-in zoneinfo module is also available. If the above failed, it would be interesting to see what result the following gives in that case.
from dateutil.parser import isoparse
from zoneinfo import ZoneInfo

start = isoparse("2023-06-26 00:00:00")
start
print(start)

influxtz = ZoneInfo("Australia/Sydney")
influxtz
print(influxtz)
type(influxtz)

start = start.replace(tzinfo=influxtz)
start
print(start)

When originally writing the script, I tested the zoneinfo module vs. dateutil and found issues with zoneinfo, so I did not use it.

Hi @mcbirse , yes, I am running the script on a Synology NAS.
The results of the first script are as follows:
2023-06-26 00:00:00
tzfile('/usr/share/zoneinfo/Australia/Sydney')
2023-06-26 00:00:00+00:00
The second script was a bit more fiddly.  The version of Python 3 currently installed in the Synology DSM is 3.8.15, so ZoneInfo is not available.  As noted above by @clukas1967, you can install Python 3.9, but it does not include PIP, and installing it is...beyond my abilities, so I could not install python-dateutil in Python 3.  I ended up installing backports.zoneinfo and updated the second line in your second script to
from backports.zoneinfo import ZoneInfo
The results of the modified second script are as follows:
2023-06-26 00:00:00
Australia/Sydney
2023-06-26 00:00:00+10:00
So ZoneInfo looks like a winner for Synology NAS users.  I will modify my version of tesla-history.py to use ZoneInfo, but it will be referencing backports.zoneinfo, so it will have to be a special 'Synology NAS users' version of the script.

Interesting. Did you get any error from Tesla when you tried the login, or just the expected 404 page URL?

I got nothing.. got no option to log in.

no option to log in.

@simonrb2000  - Just to make sure we aren't missing anything, did you follow the login instructions here? https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/tesla-history#setup-and-logging-in-to-tesla-account

Great tool! How did I not know about this? I've been using Powerwall Dashboard since January but I've had the system much longer, so getting the old data in the dash is amazing. Thanks!
Just one question: Is there any way to fetch historical SOC for each Powerwall? One of mine is reading much lower than it should (but not warranty-claim low yet - 10.4kWh) and some history on it would be great.

Hey @ceeeekay - Glad you found the tool and find it useful!
It's a godsend to be honest and was well worth the time spent developing and testing, and I still use it frequently myself... typically due to outages when you need to retrieve some missing data.
Unfortunately, as far as I'm aware there isn't any way to retrieve what you are after, which I'm guessing you mean the Powerwall Capacity (full pack energy) of your PW's.

It's a godsend to be honest and was well worth the time spent developing and testing, and I still use it frequently myself...

100%! ❤  Same here. Thanks @mcbirse 🙏

@mcbirse - thanks. I sort of assumed it wouldn't be available if it wasn't already included, but you never know for sure.
@jasonacox - I wonder if it's worth having this automatically run with a ~1 day history after an upgrade or restart? I've always been aware I'm creating gaps whenever I upgrade the dash, or patch & reboot the host, and didn't know I could recover the data this easily.
Also I feel like this tool should be mentioned on the main README.md. I only found it because I was digging around in the install directory. For new users it would be nice to start out with a bunch of history already in the DB.

++1 on this suggestion.
Been running PWD now for quite some time, and there are multiple days per month (lets say 3 days on average which is 10% of a month so significant) where there is some hiccup between PWD and the Tesla GW.
In the last week alone this happened twice.  The first “outage” was ~5 hrs and the second was ~4.5 hrs.  I’ve seen over 12 hrs - See red circles in plot below.   And in one case 2 months ago my Tesla GW lost its mind completely and the network connection had to be reprovisioned. I don’t check PWD every day, so I didn’t catch it for over a week.  I can’t be the only one having these issues.
@.***
As previously discussed on this thread, my PWD instance runs on a Synology DS1614+ with 10Gbps fiber connection to my core switch and backup power for both.  So I can be 100% confident that the Synology box is not the problem.  Ergo….
A feature that would “close” such gaps automatically would be a truly welcome and nice peace of mind, esp since the code for historical grabbing already exists and is tested.  The basic problem is the current architecture of PWD is essentially stateless.  When it queries, it logs whatever it gets for that instant in time.  If it gets nothing, then nothing happens and there’s no trigger for a future query.
The way I would craft the RFE would be like this:

PWD keeps running tally of last successful GW query w/valid returned data.  (A simple DB query should be sufficient for this, or it could be a separate running register.)
For each future GW query loop…
*   Pull current data query.  If no valid returned data, exit  (same behavior as now)
*   If valid data received, check to see delta-T between now and last valid data
*   If delta-T > 1 polling interval then enter recovery subroutine to query for all intermediate data now that connectivity has been restored

Subroutine sequentially moves through all query timepoints with null data in DB and attempts to requery.  If valid data received, overwrite null data in DB.
I’m assuming there are empty rows in the DB for the missing intervals.  If not then I guess question is can one retroactively insert data into a TSDB?

If one wished to bound this activity, the “lookback” could be limited to some hard limit vs. just the previous good data timestamp.
*   In this case I’d recommend 12 or 24 hours as lookback limit
*   Not sure why you’d “cap” the lookback period if you have a known last good data timestamp and the code has the ability to query arbitrarily far back in the past.
Obvious exception conditions to manage:
*   PWD 1st time ever startup

After first ever coldstart of PWD there will obviously be no valid previous history timestamp.
So if this timestamp is blank, obviously there would be no lookback.

 *   PWD warmstart after server outage  (e.g. server down for code upgrade or other maintenance, or Docker failure)
    *   Have to decide how to handle this.  If the “last good timestamp” suggested above is stored in the DB then this can be checked on a warmstart and treated like any other temporary GW outage scenario.
    *   I had this exact scenario a few months ago when I reorganized the disks on my Synology and had to move the whole Docker instance with PWD to another drive and it took several days to figure out how to do it successfully.
 *   PWD warmstart from backup
    *   I am now doing standalone backups of my PWD + Docker folder trees after discovering that Synology Hyperbackup does not get them.
       *   For other Synology users – if you are “checking” the “Docker” app in the backup menu all it is getting is the app **NOT** the data subdirectories
    *   Never had to do this yet, but let’s say I had a crash and had to restore a previous PWD backup.
    *   Think as long as the Docker, PWD, etc. were all properly reinstalled in the right places, then PWD would just see this as a warmstart as above.  But I flag the use case in case the internal architecture somehow would treat it differently.

This approach minimizes the brain surgery and keeps the overall stateless model while imposing a minor and simple compare with ability to conditionally execute a finite number of additional stateless queries under a single exception condition.
Regarding the issue with long term GW unreachability for unknown reasons, I would also add a second RFE to automatically send an administrator email alert every 12 or 24 hrs that no data has been received.
-cl
From: Chris @.>
Sent: Sunday, September 15, 2024 6:04 PM
To: jasonacox/Powerwall-Dashboard @.>
Cc: Chuck @.>; Mention @.>
Subject: Re: [jasonacox/Powerwall-Dashboard] Any way to import all historical data? (Issue #12)
@mcbirsehttps://url.emailprotection.link/?buEq4eyNl9NRYRhKUfm4VcIXnQvQKXIpCFfjlfOsV1q63OaLvHn_KhEd_Bevhc7IBB9u36YB9udLIqiAieUX0Lw~~ - thanks. I sort of assumed it wouldn't be available if it wasn't already included, but you never know for sure.
@jasonacoxhttps://url.emailprotection.link/?buEq4eyNl9NRYRhKUfm4VcMwSNIehMFBNnvjb-DyNv8bvm7mbRwtn7rSZcXdsrQB5hIGPOTdmiYHogCpOQU4Epg~~ - I wonder if it's worth having this automatically run with a ~1 day history after an upgrade or restart? I've always been aware I'm creating gaps whenever I upgrade the dash, or patch & reboot the host, and didn't know I could recover the data this easily.
Also I feel like this tool should be mentioned on the main README.md. I only found it because I was digging around in the install directory. For new users it would be nice to start out with a bunch of history already in the DB.
—
Reply to this email directly, view it on GitHubhttps://url.emailprotection.link/?b9Ve3c_HooY3pAjupsAXKS763DBPxxTPJgbqiADMLP8as5cAojxTufUVdYRJsV6AfwBV3eoCykiswH8-2PD5BvohAreZpWNc5q_YMTt4pmfzSXcZ0ir8nmErwqUzmXhAvEEeLxTd1oleZlQ6QWGBvATm8prcDrNyPryej_NAQnYk~, or unsubscribehttps://url.emailprotection.link/?bGcW7RWEOBEExltVFNdoiAzuSLb_XtNqf441768nRCBoMEe4FHMWPuTf-aDu-QphUZOt11IPYUtgJq9yNAJ9nce1r8d6O7J9GdExPblT6pfgraJsyQRimM41ZzSR4lMsEcvsku8i93wYtnJzAGunsZUS06We32eVmGrewlp8mLNHSMiSmZL8IQWvzcoWZ9jDZ7J7_ajB469xCdPaMV3-HgC3DYv7W-6ADEJywcK58yAIVAYnf6gNoJZ7meI_5AOgw.
You are receiving this because you were mentioned.Message ID: @.***>

Here is the image that goes with the comment I just made showing two different days in the last week that I had a 4-5 gap overnight in the data.

I like the idea of having the history import at setup time (and update during upgrades). It would require that the user signs in to the Tesla cloud which would create another set during setup. It probably makes sense to make it an option.
TODO

Hi Thank you for this. I have go the dashboard up and running and would like to import my historical data. However how do I install the required python modules: pip install python-dateutil teslapy influxdb when i run this I get the error "error: externally-managed-environment"
I have on another linux VM run the https://github.com/netzero-labs /tesla-solar-download

The "externally-managed-environment" error typically occurs when you're trying to install packages in a virtual environment that's not properly activated.
To resolve this, try activating your virtual environment before running the pip install command. If you're using a Python environment like conda, you can activate it with conda activate <env_name>. If you're using a virtual environment created with python -m venv, you can activate it with source <env_name>/bin/activate (on Linux/Mac) or <env_name>\Scripts\activate (on Windows).


Access denied for user graphite to database graphite on Graphite's web ui

After an VM cluster is built from uefi_support, the Graphite's web ui shows a green error message.
"OperationalError: (1044, "Access denied for user 'graphite'@'%' to database 'graphite'")"

I was able to re-create this issue.  The cause is that we attempt to use the database name graphite.* in a mysql_database resource, when we should just be using graphite.  It's fixed.


Adds command pp solution get, closes #3722

This PR adds the command pp solution get, which will retrieve a specific solution in a given Power Platform environment.
Closes #3722

Thank you @nicodecleyre
We'll review it ASAP!

Hi @nicodecleyre
Seems like a force push from our side kinda messed op your PR. Could you resolve this please?
Best way to do this is by doing the following:

Sync all new changes to your local main branch
Switch to your feature branch card-get
Run the command git rebase main
(Resolve merge conflicts if needed)
git push

When this has been done, you can mark this PR as 'ready for review' so we can do a review.
Thanks!

Thx to @milanholemans  for the help when mergin the messed pr

although the same querystring is used for getting the solution by name & by id, there is an extra parameter in the response when getting the solution by id _organizationid_value

although the same querystring (with select parameters) is used for getting the solution by name & by id, there is an extra parameter in the response when getting the solution by id _organizationid_value

I'll have a look at it while reviewing. If it's an API behavior then we'll just leave it as is.

Hi @nicodecleyre
I'm having trouble merging your PR, as you can see it contains a commit from Reshmee, also every line from the TOC is edited by you. Could you have a look at it? Worst case you can create a new clean branch with a new PR.


chore: make a small change to test approval permissions

Test purposes only - making a small change to test PR approval process for external collaborators.
Checklist:

[ ] I have updated the necessary documentation
[ ] I have signed off all my commits as required by DCO
[ ] My build is green

Waiting for @ritikesh - @procore-oss/procore-blueprinter - please do not approve.


wrapcontainer css 수정, 레이아웃 수정, 롤업 delete 추가(svg 대응)

:tada: This PR is included in version 1.3.11 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Upgrade spring-security-oauth2-autoconfigure ver

Upgrade spring-security-oauth2-autoconfigure version to 2.4.5

we're unable to update to minors in a patch release.


Handle relative schema paths. Fix #47

It is a simple change, but I have to pass cwd to all recursive xsd2py.schema_to_py stack.
This change support things like:
# wsdl
<xsd:import namespace="http://example.com/1" schemaLocation="one/1.xsd"/>

# 1.xsd
<xsd:import namespace="http://example.com/2" schemaLocation="two/2.xsd"/>

# 2.xsd
<xsd:import namespace="http://example.com/3" schemaLocation="../three/3.xsd"/>

I think in Python 3 you get need to deal with byte vs. str. For the 2.6 failure you can use "assert_contains" from pythonic_testcase.

I fixed the tests. Travis is good now.


The contact is not saved if it has been deleted from the address book

Steps:

Open the address book\contacts
Copy the address of the contact
Delete contact
Open the wallet tab
Open the address book\contacts
Try adding the copied contact
Expected: The contact is added to the address book
Actual: The contact is not added to the address book

Additional context
If you close the app and reopen it, the contact you added is displayed

Checked


How to test with header-based API versioning?

Hi,
I'm interested in using the new header-based API versioning, but I'm unsure how to test this using the suggested RSpec Request specs. The normal RSpec way of setting HTTP headers is to set the header on the rack request object, like request.env["HTTP_ACCEPT"] = "application/vnd.twitter-v1+json".
But RSpec request tests don't allow you to set the HTTP header in this way – the request object is nil. Any ideas?

I assume you're probably using Rack::Test and have Rack::Test::Methods included into your example groups.  If that's the case, then the get put post delete methods actually take a 3rd parameter what are the headers:
# in spec helper
RSpec.configure do |config|
  config.include Rack::Test::Methods
end

describe 'versioned get' do
  def app
    # your rack app here
  end

  it 'should using HTTP_ACCEPT header for versioning' do
    # the 3rd parameter is any http headers you want to add
    get '/hello', {}, {'HTTP_ACCEPT' => 'application/vnd.twitter-v1+json'}
    last_response.status.should == 200
  end
end

You can also look at the spec tests for grape for an example of how I tested that.  spec_helper.rb and shared_versioning_examples.rb are good starting points

Perfect, thanks! I was using Rails/Rspec's built-in test helpers, and RSpec request tests don't allow the same fine-grained control that Rack::Test provides. I'm all set now, thanks for your help!


Allow overriding DOCKER_HUB_REPO

Allow the user to change DOCKER_HUB_REPO by --docker-hub-repo instead of modifying something like ~/.local/lib/python3.?/site-packages/bugswarm/common/credentials.py
Close #7

I think we need it when we need to use an experimental repo or the cached repo (bugswarm/cached-images). Currently the only workaround I know is #7, but I hope this can be a better way.

I think we need it when we need to use an experimental repo or the cached repo (bugswarm/cached-images). Currently the only workaround I know is #7, but I hope this can be a better way.

What do you mean an experimental repo?
If you are doing some testing, you do not have to use the client. You can use Docker commands.

I think we need it when we need to use an experimental repo or the cached repo (bugswarm/cached-images). Currently the only workaround I know is #7, but I hope this can be a better way.

What do you mean an experimental repo?

Something like https://hub.docker.com/repository/docker/lxylxy123456/bugswarm-repo , which was created when I was running the bugswarm pipeline.

If you are doing some testing, you do not have to use the client. You can use Docker commands.

Just to clarify, are the following two commands equivalent?
$ bugswarm run --image-tag apache-commons-lang-159112598 --docker-hub-repo bugswarm/cached-images
$ docker run --rm -it bugswarm/cached-images:apache-commons-lang-159112598

I think we need it when we need to use an experimental repo or the cached repo (bugswarm/cached-images). Currently the only workaround I know is #7, but I hope this can be a better way.

What do you mean an experimental repo?

Something like https://hub.docker.com/repository/docker/lxylxy123456/bugswarm-repo , which was created when I was running the bugswarm pipeline.

If you are doing some testing, you do not have to use the client. You can use Docker commands.

Just to clarify, are the following two commands equivalent?
$ bugswarm run --image-tag apache-commons-lang-159112598 --docker-hub-repo bugswarm/cached-images
$ docker run --rm -it bugswarm/cached-images:apache-commons-lang-159112598

Yes those commands should be equivalent.

Closing this issue because there is a workaround.


[Shopify] - Add Item As Variant Tests

This pull request does not have a related issue as it's part of delivery for development agreed directly with @AndreiPanko
Fixes #26819
Add Item As Variant Test
Automated tests for changes #26712
Fixes AB#555979

@petemchlk, this PR has been processed internally. Now we just need to merge this one. Could you pull from main to make CI pass please?

@JesperSchulz done :)

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: .

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 26251.


[REEF-571] Remove NuGet for Org.Apache.REEF.Bridge.JAR

This addressed the issue by

Removing NuGet package generation from Org.Apache.REEF.Bridge.JAR

JIRA:
REEF-571

I'll do a pass.

@anupam128 I've done a pass.

Addressed the comments.

@anupam128 Thanks, I will test and merge


RawPlugin: synthetic default imports

Hi, I'd like to make a PR to make RawPlugin work with synthetic default imports. I see three reasonable ways of implementing it:

add an option to RawPlugin, e.g. RawPlugin({useSyntheticDefaultImports: true})
detecting the configuration (allowSyntheticDefaultImports in tsconfig.json or an option passed to FuseBox.init)

May I ask someone more experienced with the codebase for an opinion?

@merlinnot we can detect allowSyntheticDefaultImports that too.
https://github.com/fuse-box/fuse-box/blob/master/src/core/TypescriptConfig.ts object is available in context (which has all the config loaded)
Also it would be nice to introduce an option.
Basically we check for it here:
https://github.com/fuse-box/fuse-box/blob/f680f49b7e7ceefd1387c9a6426155e393fac0fd/src/plugins/RawPlugin.ts#L73-L77
And do similarly to HTMLPlugin
https://github.com/fuse-box/fuse-box/blob/f680f49b7e7ceefd1387c9a6426155e393fac0fd/src/plugins/HTMLplugin.ts#L61-L65

Isn't it inconsistent with CopyPlugin? https://fuse-box.org/page/copy-plugin#usage

This has been all fixed in v4. Please upgrade to the major version


Actors with the same sprite take up multiple frame slots

Describe the bug
Adding an actor with a sprite already used by another actor doubles the amount of frames used. Adding another copy triples it. This is not just a counting error in the IDE. Going past the sprite limit this way creates graphical glitches
Expected behavior
If actors share the same sprite sheet, the same sprite tiles should be used in memory
Screenshots
https://user-images.githubusercontent.com/57502156/181091167-d619ecbd-7d12-4fdb-b2c5-2809a0d137f8.mov
See video here.

Platform (please complete the following information):

OS: MacOS 10.14

Additional context
Memory use and sprite states is confusing and undocumented

Actors that don't use "set sprite sheet" on any of their scripts will be share the same sheet. However, in order to use "set sprite sheet" on an actor, their tiles need to become unique in memory, otherwise replacing the sprite sheet for that actor would also replace it for others using the same sheet initially.

Actors that don't use "set sprite sheet" on any of their scripts will be share the same sheet. However, in order to use "set sprite sheet" on an actor, their tiles need to become unique in memory, otherwise replacing the sprite sheet for that actor would also replace it for others using the same sheet initially.

Do you have expected use cases for these specifics? It’s pretty hard to wrap my head around

The unique sprite tile count can be seen in the sprite editor and scene, this is how much vram a sprite, or all sprites are expected to allocate.
Set animation states:
Allocates all unique sprites tiles to vram, will not modify vram mid game, faster and more flexible in certain cases.
Actors can reuse this sprite.
Player is reserved separately (as another scene may change the player character.)
Animation states like "death" as part of the enemy sheet can be reused between actors.
Set Sprite Sheet:
Allocates as much vram as the largest sheet for this actor, modifies vram mid game.
Marks the sprite as "Exclusive" uniquely reserved vram, to prevent other sprites from being modified.
Actors that change sprite sheet will no longer be able to reuse sprites.
May show a graphical glitch over 1-2 frames as vram updates before metasprite frame.
Use case ideas for 3.0 "set sprite sheet"
Weapon switching or rare special player attacks that would take a lot of vram in 1 sheet,
A large boss sprite with animations that take too much vram.
1-2 Turn based battle sprites, Switch to one of many enemies within the same scene (instead of separate scenes), as long as there is enough vram for 2 copies of 1 sprite (impossible in 2.0 where all sprites were pre-allocated)
Use case ideas for 3.0 "set anim state"
Reused actors, "death" or "explode" state, instant changes in animation, special gameplay poses like attacking, crouching, flashing damage, run/walk speeds via doubling frames, Any special or cutscene animation with per pixel sprite placing movement (rather than more sprites).
Palette slot changes per state, like a red palette reserved on slot 2 for actors taking damage.

Extra notes:
-Animation states are similar to 2.0's pre allocation of sprites, when vram could not change mid game for actors, states are manually set up with the sheet. You could combine all enemy actors into one sheet with states, including the explosion/death frame if you are extra tight on vram.
-You may have 255 max 'anim frames' for a sprite sheet, spread across directions and animation states,
-Try to use generic reusable state names across sprites, to reduce custom state definitions in ram/while scripting.
-3.0.3 note that emote seems to always use vram tiles 63-64, may cause problems when trading background vram tiles for extended sprite tiles 65-96.


Remove about from navbar in all the features page #104

issue #104
I have done this and remove about from navbar in all the features page #104
about navbar is now only on the index.html page, I have removed "About" line of code from every feature pages.

were u assigned this issue? i think some other person was assigned @mnik7044

I am novice here so don't know about assign things, so now it is possible to assign this to me.

I am working on this issue @ankitxcoding


cloudtasks: initial commit

Note: you must be whitelisted to use this API.

Taking this one over per discussion with @broady

This PR is deprecated


2x2 void

Why 2x2 incinerator
Incinerator is item void but needs power
Just place more incinerator

Oh no
https://www.reddit.com/r/Mindustry/comments/npyago/should_upgraded_factories_have_a_2x2_incinerator/


Update Python travis build

Python build is breaking after a Puppeteer update that makes it incompatible with its VM's version of Node. This PR uses a more recent version of node.

Background

Change List

Coverage increased (+2.7%) to 82.884% when pulling 0681c8b5653dc880e7e7af86539c4b01e7005015 on ajd/update-puppeteer into 49d53eadb77889a1b662457f09f9cb0204f020e1 on master.


README.md: Add getting started example

I've spent more time than necessary getting started due to the lack of documentation. Hopefully this will help others avoid the same. I'm sure the changes here could be improved, so I'm open to suggestions.

Hi,
Thank you for your contribution. This seems to solve a specific (but common) problem, so I will probably include it in the wiki.

bump

I have no plans to continue this.


🐞 Updating app config from server plugin isn't get carried to client app

Environment

Operating System: Darwin
Node Version:     v18.15.0
Nuxt Version:     3.5.2
Nitro Version:    2.4.1
Package Manager<EMAIL_ADDRESS>Builder:          vite
User Config:      app, appConfig, typescript, build, devtools, vite
Runtime Modules:  -
Build Modules:    -

Reproduction
https://stackblitz.com/edit/github-fwi5t5
Describe the bug
Hi 👋🏻
I've very special requirement and for that I have to update the appConfig (e.g. theme) from server (I choose plugin for this work) based on visited URL (I'll be serving same nuxt app to multiple subpaths for demo purposes using nginx).
When I try to set the appConfig from the server plugin it updates it but as soon we reach to the client app appConfig gt reset 🤔 . For example if set theme to dark using server plugin then I dont get dark on the client side.
Additional context
More context on what I'm doing. We building admin templates and we are adding nuxt support. We display 6 demos for our template and each has different subpath based on demo number (/vuexy-vuejs-admin-template/demo-1,2,3,4,5,6). We can't run 6 instances of nuxt/node app and trying to server single nuxt app via nginx when these six URL is visited.
If you have any advice on achieving above scenario let me know, I'm still exploring.
Logs
No response

Unlike runtime config, app config is not included in the payload. So you could set a key there with any changes you might want to make, e.g. nuxtApp.payload.updatedConfig = {} and then merge that into the appConfig on client-side.
We might consider doing this automatically for better DX...

We think it would be a good idea to create a smart proxy for app config + runtime config to avoid cloning it unless needed, and this could also handle automatically including changes in payload.

@danielroe I'm trying to do the same thing and I'm not sure what's the best way to go about this. I want to get the colorMode cookie as soon as possible to prevent "flashing" light mode if dark mode is wanted.
I tried to do it in a nitro hook on the server, but it didn't work.  I could read app.config.ts, but I couldn't edit it.
What would be the best (earliest ?) time/hook to do important tasks like getting color mode, auto-login the user, etc ?
My solution for now is the app:created hook.  But it runs twice, once on the server, once in the client.
Is it fine to let it run twice ? or is it redundancy and a loss of time ?
Would it be beneficial to handle color mode on the server ? or is it fine to handle it on the client? Too early ? too late?
( by the way, you did a great job on the Mastering Advanced Nuxt.js video !! )

@danielroe I'm trying to do the same thing and I'm not sure what's the best way to go about this. I want to get the colorMode cookie as soon as possible to prevent "flashing" light mode if dark mode is wanted. I tried to do it in a nitro hook on the server, but it didn't work. I could read app.config.ts, but I couldn't edit it.
What would be the best (earliest ?) time/hook to do important tasks like getting color mode, auto-login the user, etc ? My solution for now is the app:created hook. But it runs twice, once on the server, once in the client. Is it fine to let it run twice ? or is it redundancy and a loss of time ? Would it be beneficial to handle color mode on the server ? or is it fine to handle it on the client? Too early ? too late?
( by the way, you did a great job on the Mastering Advanced Nuxt.js video !! )

@Posoroko I see you’re having a tough time with this issue. I’d be happy to help you out in detail. If you’re comfortable, we could hop on a Discord call to walk through the problem together. Let me know if that works for you
https://discordapp.com/users/224828877082460160


chore: stop using macos CI runners

In the past, we had issues with ubuntu runners in certain situations and were forced to use macos, which is much more expensive. Those issues have now been resolved on the Azure/github side, so we can switch back to ubuntu to reduce costs.

Oh nice!


Improve docs for versioning options, and add new path-based support.

Signed-off-by: Dane Jensen<EMAIL_ADDRESS>

LGTM