Need accurate fuel gauge for use with OW+ with Two-X upgrade

Kevin and I were talking about how those of us with Two-X OW+s could have accurate remaining battery indication.
The Two-X replaces 2.5Ah OW+ battery with 5Ah battery otherwise the same.  But it seems the board insists on assuming it has 2.5Ah battery.  Symptom is, after full charge, app says 100%.  Say you use up 1.25Ah, then the app will say 50%, as if it had a 2.5Ah battery still.  But with Two-X that should be 75%.
Kevin suggested (and I am fricken all about) putting a settings switch for Two-X users that corrects the gauge somehow.  Is it possible to accurately estimate battery remaining with the info available to us?

Thanks!
Soren
<EMAIL_ADDRESS><EMAIL_ADDRESS>

My OW is broken in that it shows 100% for miles. I'd be all for an implementation that tries to gauge based on voltage. Maybe storing the min and max known voltages in sharedPrefs? Or just guessing on the min voltage?

Hey all. I've been estimating remaining range by the voltage with my 2X CnR
setup using the EGO battery directly connected through the XLR and the
existing internal battery. My voltage chart is not completely accurate but
better than every app I've used including Ponewheel. I was going to look up
the code to see how Ponewheel calculates range remaining and battery % but
haven't yet.
Attached is the chart.
On Fri, Jul 13, 2018, 7:14 PM Erik Babel<EMAIL_ADDRESS>wrote:

My OW is broken in that it shows 100% for miles. I'd be all for an
implementation that tries to gauge based on voltage. Maybe storing the min
and max known voltages in sharedPrefs? Or just guessing on the min voltage?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ponewheel/android-ponewheel/issues/76#issuecomment-404983537,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APHfOWuElGUPpLoogNWkLuL8_ikTBbDBks5uGTfpgaJpZM4VPpV_
.

@muellergit attachment didn't work via email. Care to add directly on github?

@muellergit
The %/State of Charge comes directly off the OW (0-100). I assume it keeps an internal voltage chart, that gets out of line occasionally and needing a reset by draining the battery and nosediving on hillway ave.
Thoughts on how we can extend the model you have to any battery source? From what I can tell on https://en.wikipedia.org/wiki/State_of_charge this would be an "offline" technique reading the battery current comparing it over time w/ a hash/lookup table. The table would be the model, reset manually or when the board is 100%. Something like that. We could add another "estimated/smart/extended battery %" metric having an option to making that the primary source of truth for battery % off the board.

A voltage lookup table would be a really nice option which should work with
small and large batteries.  This would probably work fantastic, and keep us
from having to mentally linearize the the voltage level based on the
discharge curve.  But it's dependent on the shape of that curve, which
differs with battery chemistry and with degradation level.
Could it somehow self-calibrate by counting the charge used, and then use
that number for future estimates?  This is probably too tricky to do
reliably, but would be so cool if it worked, because it would not only
adjust to people changing their battery setup, but also it would stay
accurate as a battery degraded over time.
What do you guys think?  We all appreciate this so much!
Thanks,
S
On Sat, Jul 14, 2018 at 10:40 AM, Kevin Watkins<EMAIL_ADDRESS>wrote:

@muellergit https://github.com/muellergit
The %/State of Charge comes directly off the OW (0-100). I assume it keeps
an internal voltage chart, that gets out of line occasionally and needing a
reset by draining the battery and nosediving on hillway ave.
Thoughts on how we can extend the model you have to any battery source?
From what I can tell on https://en.wikipedia.org/wiki/State_of_charge
this would be an "offline" technique reading the battery current comparing
it over time w/ a hash/lookup table. The table would be the model, reset
manually or when the board is 100%. Something like that. We could add
another "estimated/smart/extended battery %" metric having an option to
making that the primary source of truth for battery % off the board.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ponewheel/android-ponewheel/issues/76#issuecomment-405038568,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AnTpW5EGYPGKl3RtbPnUJx76pGCngg17ks5uGi0NgaJpZM4VPpV_
.

I'm still thinking about this. I have started riding with 28 psi instead of
22 psi because I want to see how significant tire pressure is on the range
estimate. After 1 ride at 28 psi the difference is inconclusive. I actually
got less range with higher pressure but went on a new route with huge hills
and a large grass section.
Another tidbit I just figured out is that the power button pulsing that
indicates charge level still works without the BMS connected and also when
the app is showing that the battery is empty but you are using CnR or have
more Ah in parallel. So the power button pulsing seems to be based on
battery voltage and not counting Ah.
Are any of you on the OW Mad Scientist Facebook group? I've had some
discussions there about this problem. I know Soren (Two-X) is on it.
On Sun, Jul 15, 2018, 9:31 AM two-x<EMAIL_ADDRESS>wrote:

A voltage lookup table would be a really nice option which should work with
small and large batteries. This would probably work fantastic, and keep us
from having to mentally linearize the the voltage level based on the
discharge curve. But it's dependent on the shape of that curve, which
differs with battery chemistry and with degradation level.
Could it somehow self-calibrate by counting the charge used, and then use
that number for future estimates? This is probably too tricky to do
reliably, but would be so cool if it worked, because it would not only
adjust to people changing their battery setup, but also it would stay
accurate as a battery degraded over time.
What do you guys think? We all appreciate this so much!
Thanks,
S
On Sat, Jul 14, 2018 at 10:40 AM, Kevin Watkins<EMAIL_ADDRESS>wrote:

@muellergit https://github.com/muellergit
The %/State of Charge comes directly off the OW (0-100). I assume it
keeps
an internal voltage chart, that gets out of line occasionally and
needing a
reset by draining the battery and nosediving on hillway ave.
Thoughts on how we can extend the model you have to any battery source?
From what I can tell on https://en.wikipedia.org/wiki/State_of_charge
this would be an "offline" technique reading the battery current
comparing
it over time w/ a hash/lookup table. The table would be the model, reset
manually or when the board is 100%. Something like that. We could add
another "estimated/smart/extended battery %" metric having an option to
making that the primary source of truth for battery % off the board.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
https://github.com/ponewheel/android-ponewheel/issues/76#issuecomment-405038568
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AnTpW5EGYPGKl3RtbPnUJx76pGCngg17ks5uGi0NgaJpZM4VPpV_
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ponewheel/android-ponewheel/issues/76#issuecomment-405095906,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APHfOeqtp4_Wv2M0s-Vs28-Bpr8blcb6ks5uG1IlgaJpZM4VPpV_
.

I don't think that's right, that the blue light is based on voltage.  When the Two-X is at half-charge, and the app says 1%, the blue light also blinks like 1%.
How about a "auto-calibrate fuel gauge" switch in the app, where, if it is turned on, then every time it hits full-charge, it begins an energy counter.  Then it counts Ah in either direction as you use it. If a charger is connected before the board is fully drained, then it's a no-go, it starts over once it hits 100%.
But if the board is ridden till it dies, then it has a complete count of Ah in the battery.  It can store that until the next full cycle.  The fuel gauge can base its percentage off counting Ah against this.
The problem I see is the fuel gauge only works if the app is always connected.  Unless the board reports Ah used?  Does it?

The alternative of course is a lookup table to linearize the voltage reading into a linear fuel gauge.  This would be pretty easy except the voltage is also affected by loading of the battery.  And given different batteries will skew the voltage differently, now this is a 3-dimensional lookup table or a lookup table with a parameterized post-processing formula.  All of a sudden it's not as easy.
But if it could be made to work then it solves the problem of a fuel gauge for alternate battery systems, and also automatically adjusts to stay accurate as a stock battery degrades with use, and wouldn't require the app to stay connected.
I wonder what other ideas people might have?  Would it help if I made a ponewheel log of a full discharge/charge cycle on my two-x board for you?
Thank you so much!
Soren

Thanks, I don't think you are being preachy.
I was thinking it would need to be a 4 dimensional lookup table already to
include different curves for different temperatures and for different
battery chemistries. But it is also possible that some of these lookup
parameters may not really add that much to the accuracy of the estimate
anyway, because of the unknowns about the conditions you will be riding on
the remaining ride are probably always more significant than some of the
parameters make.
Instead of using the usage and regen stats at all, I am interested in
exploring if battery voltage alone can work. My simple chart already works
better for me even with a completely stock OW+.
Actual range for each trip would be compared to the lookup table's expected
range as the trip progresses and an adjustment factor would be calculated
and applied. This would help compensate for riding on street versus grass,
high tire psi and low psi, light weight rider versus heavy rider, etc.
On Tue, Jul 24, 2018, 6:25 PM two-x<EMAIL_ADDRESS>wrote:

I'm sorry, I don't mean to sound all preachy. I think what I'm proposing
is very much like what you described, muellergit.
Let me know if I can help - I can log data, do research, or whatever.
S
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ponewheel/android-ponewheel/issues/76#issuecomment-407583700,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APHfOTvh4EYVnBJyqOhbTVKpW0hidLraks5uJ6zWgaJpZM4VPpV_
.

I thought of another related feature while riding today that we should look
into: monitoring voltage for motor load induced sag and sending an audio
notification in the app when it exceeds some threshold before cutting out.
This would help new riders in identifying pushback if there is a
correlation between pushback and a certain amount of voltage sag.
On Jul 25, 2018 12:00 AM, "Mike Mueller"<EMAIL_ADDRESS>wrote:
Thanks, I don't think you are being preachy.
I was thinking it would need to be a 4 dimensional lookup table already to
include different curves for different temperatures and for different
battery chemistries. But it is also possible that some of these lookup
parameters may not really add that much to the accuracy of the estimate
anyway, because of the unknowns about the conditions you will be riding on
the remaining ride are probably always more significant than some of the
parameters make.
Instead of using the usage and regen stats at all, I am interested in
exploring if battery voltage alone can work. My simple chart already works
better for me even with a completely stock OW+.
Actual range for each trip would be compared to the lookup table's expected
range as the trip progresses and an adjustment factor would be calculated
and applied. This would help compensate for riding on street versus grass,
high tire psi and low psi, light weight rider versus heavy rider, etc.
On Tue, Jul 24, 2018, 6:25 PM two-x<EMAIL_ADDRESS>wrote:

I'm sorry, I don't mean to sound all preachy. I think what I'm proposing
is very much like what you described, muellergit.
Let me know if I can help - I can log data, do research, or whatever.
S
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ponewheel/android-ponewheel/issues/76#issuecomment-407583700,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APHfOTvh4EYVnBJyqOhbTVKpW0hidLraks5uJ6zWgaJpZM4VPpV_
.

I have created pull request #92 to resolve (or at least make a first effort resolve) this issue.

Great stuff from mr. @biell! Did some basic smoke tests and looks to be solid enough for master. As soon as we nail down the stability issue (https://github.com/ponewheel/android-ponewheel/issues/91) will roll into production.
@two-x to confirm before closing after we get him a solid release :)

You can check out the latest attempt, merged into master.  This new method carries with it improved voltage calculations and amp draw filtering (adding up the voltage values of each cell is proving more stable than using the reported total voltage).  However, that is not the exciting part.  The Two-X method (which assumes that each time you charge your OneWheel, you charge it all the way) will keep track of your amphr usage across a long ride, figure out how many amphrs / remaining% there are and keep that progression going.  This is proving very accurate and I think everyone will be happy with it.  The only rub is that, if you turn your OneWheel on with less than 60% remaining, it doesn't have enough history to perform the calculations and has to fall back on reported cell voltages.  I have ridden from 100% down to 1% reliably over a dozen times in testing.
I have about 40 charge cycles on my Two-X, anyone with more care to comment on how this might affect my math?  What I am most interested in is what is the real percent battery left when the OW thinks it is at 1%.  Will that stay around my personal value of 55% Two-X remaining, or will it slip closer to 50%.
At any rate, we are making progress!
You can review the math in the source tree: app/src/main/java/net/kwatts/powtools/util/Battery.java

I was able to try the latest version available from the Play store and was
hoping that one of the options such as "Output voltage" might work with my
Zombied Plus (running without the BMS). The voltage is still available in
the apps without the BMS, but the Ponewheel app shows 101% for all of the
battery setting options.
Could the your method based on the Output voltage be made to work for
anyone who is no longer using the BMS? It's basically what I use manually,
but I need to stop to let the voltage settle at no load to get a good idea
of charge level based on voltage.
Right now my Plus voltage is 52.4 in Ponewheel but it shows 101% battery,
no matter what I have the battery method set to.
Thanks for all the work!
On Tue, Oct 15, 2019, 3:46 PM William Totten<EMAIL_ADDRESS>wrote:

You can check out the latest attempt, merged into master. This new method
carries with it improved voltage calculations and amp draw filtering
(adding up the voltage values of each cell is proving more stable than
using the reported total voltage). However, that is not the exciting part.
The Two-X method (which assumes that each time you charge your OneWheel,
you charge it all the way) will keep track of your amphr usage across a
long ride, figure out how many amphrs / remaining% there are and keep that
progression going. This is proving very accurate and I think everyone will
be happy with it. The only rub is that, if you turn your OneWheel on with
less than 60% remaining, it doesn't have enough history to perform the
calculations and has to fall back on reported cell voltages. I have ridden
from 100% down to 1% reliably over a dozen times in testing.
I have about 40 charge cycles on my Two-X, anyone with more care to
comment on how this might affect my math? What I am most interested in is
what is the real percent battery left when the OW thinks it is at 1%. Will
that stay around my personal value of 55% Two-X remaining, or will it slip
closer to 50%.
At any rate, we are making progress!
You can review the math in the source tree:
app/src/main/java/net/kwatts/powtools/util/Battery.java
https://github.com/ponewheel/android-ponewheel/blob/master/app/src/main/java/net/kwatts/powtools/util/Battery.java
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ponewheel/android-ponewheel/issues/76?email_source=notifications&email_token=ADY56OPJQT3O2ICFFT67IG3QOY2Z7A5CNFSM4FJ6SV72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBKKNUA#issuecomment-542418640,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADY56ONJNRLPR3ND2JRNSMDQOY2Z7ANCNFSM4FJ6SV7Q
.

It should mostly work on your plus.  I don't have an XR to test with, so the curve is probably in need of tweaking for the XR.
I would think the Output Voltage and Cell Voltage method could both be tried.  I think you will find out that those two methods are highly inaccurate near 100%, and become increasingly accurate as you approach empty.  The battery simply fluctuates less at lower end voltages.  I think the Cell Voltages method works better.
You can either clone this repository and build it yourself or wait until we get my changes into the development app store.  My goal was always to get this all released and confirmed stable first, then spend some time collecting data from people to tweak the parameters for following the voltage curve and make the whole thing more accurate, including at different speeds.

Without the BMS, there are no cell voltages so that one won't work. If the
output voltage method it's using BMS data that is also why it doesn't work.
But the FM app still shows battery voltage on my Plus without the BMS and
so does Ponewheel.
Anyway at 52.4v battery voltage the Plus is actually at about 60% charge
but it showed 101% in every battery meter mode.
On Tue, Nov 26, 2019, 9:24 PM William Totten<EMAIL_ADDRESS>wrote:

It should mostly work on your plus. I don't have an XR to test with, so
the curve is probably in need of tweaking for the XR.
I would think the Output Voltage and Cell Voltage method could both be
tried. I think you will find out that those two methods are highly
inaccurate near 100%, and become increasingly accurate as you approach
empty. The battery simply fluctuates less at lower end voltages. I think
the Cell Voltages method works better.
You can either clone this repository and build it yourself or wait until
we get my changes into the development app store. My goal was always to get
this all released and confirmed stable first, then spend some time
collecting data from people to tweak the parameters for following the
voltage curve and make the whole thing more accurate, including at
different speeds.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ponewheel/android-ponewheel/issues/76?email_source=notifications&email_token=ADY56OKRKK3G225CB5ZY3T3QVXY67A5CNFSM4FJ6SV72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFIHVEQ#issuecomment-558922386,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADY56OMDDK2WTUBULSDJ2S3QVXY67ANCNFSM4FJ6SV7Q
.

Unfortunately, the only correct way to evaluate the State of Charge is via coulomb counting. One can't do that just using voltage, specially if there is no feedback on the load being applied at the moment of the measurement.
Therefore, one easy fix would be to swap the BMS 5mΩ shunt for a 2.5mΩ one, therefore registering only half of the current passing through (equals to doubled capacity). However, this would also mess with BMS over-current detection circuits. Most of them are there to protect the battery, and the 16S2P of the Two-X can take twice the current, so it should not be a big safety risk.
Another more involved way would be to flash the BMS PIC16F with a custom firmware, adapted to the desired capacity (as it is, battery capacity is hardwired in the code).

Without the BMS, there are no cell voltages so that one won't work. If the output voltage method it's using BMS data that is also why it doesn't work. But the FM app still shows battery voltage on my Plus without the BMS and so does Ponewheel. Anyway at 52.4v battery voltage the Plus is actually at about 60% charge but it showed 101% in every battery meter mode.

Did you build your own ponewheel from the master tree?  101% should never be displayed as the code sits now.
With the code, as it sits in the repository now, I have 52.3v set to 50% battery, so that is pretty close to what you are saying should show about 60% charge.  The Output Voltage method is the least reliable of all the methods I have worked on, but should give you an idea of what you have left as you start to drain the battery, again, becoming more accurate at lower levels to keep you from having the OW run out of battery.
I have put the least effort into perfecting this method, and so the parameters need to be tweaked, I am sure.  How amp draw and tire rotation affects the voltage output, and how to smooth that out via decaying average could all be tweaked more over time.
Of all the variables I have to work with, Output Voltage fluctuates the most, and so is the hardest to guestimate from.

@ocornu ,

Unfortunately, the only correct way to evaluate the State of Charge is via coulomb counting. One can't do that just using voltage, specially if there is no feedback on the load being applied at the moment of the measurement.

I do use amp draw in my calculations, does that come from the BMS?  If so, then I would not be able to help @muellergit, because the amp draw is a critical part of smoothing out the output voltage.

One easy fix would be to swap the BMS 5mΩ shunt for a 2.5mΩ one, therefore registering only half of the current passing through (equals to doubled capacity overall). However, this would also mess with BMS over-current detection circuits. Most of them are there to protect the battery, and the 16S2P of the Two-X can take twice the current, so it should not be a big safety risk.

You wouldn't even need pOnewheel then, right?  The regular app would mostly work.  My testing shows that, on my board, the max stock OW battery only goes to 44% of my total battery on a hot day and 46% on a cold day vs my Two-X capacity.  So, it isn't exactly 50%, but close.

Another more involved way would be to flash the BMS PIC16F with a custom firmware, adapted to the desired capacity (as it is, battery capacity is hardwired in the code).

My guess here is that a FM firmware update to the BMS allowing a maximum Amp Hours capacity of 5.5 would make the vendor app just work with a Two-X, but I could be wrong.
My primary method for a Two-X right now is my "Fully Charged Two-X" option, which is very reliable assuming each time you charge, you charge until the charger light goes green and you start your ride with 60% or more battery remaining.  Under these circumstances, I keep track of how many amp hours drawn equals what percentage of the total battery capacity, and I keep that method going for the user even after the OW things it is at 1%.  If you start a ride with less battery, I fall back to the method of using the Cell Voltages reported, which is OK for accuracy from 20-40%,  pretty good for accuracy from 10-20%, and reliable for accuracy from 1-10%.

Yeah, amp draw is from the BMS so that is why that mode doesn't work on my
Zombie. Thanks, I forgot that you had said that earlier in this thread.
I am using the Play store version but it does show 101% for all modes. It
must be due to the missing BMS data somehow.
On Wed, Nov 27, 2019, 11:32 AM William Totten<EMAIL_ADDRESS>wrote:

@ocornu https://github.com/ocornu ,
Unfortunately, the only correct way to evaluate the State of Charge is via
coulomb counting. One can't do that just using voltage, specially if there
is no feedback on the load being applied at the moment of the measurement.
I do use amp draw in my calculations, does that come from the BMS? If so,
then I would not be able to help @muellergit
https://github.com/muellergit, because the amp draw is a critical part
of smoothing out the output voltage.
One easy fix would be to swap the BMS 5mΩ shunt for a 2.5mΩ one, therefore
registering only half of the current passing through (equals to doubled
capacity overall). However, this would also mess with BMS over-current
detection circuits. Most of them are there to protect the battery, and the
16S2P of the Two-X can take twice the current, so it should not be a big
safety risk.
You wouldn't even need pOnewheel then, right? The regular app would mostly
work. My testing shows that, on my board, the max stock OW battery only
goes to 44% of my total battery on a hot day and 46% on a cold day vs my
Two-X capacity. So, it isn't exactly 50%, but close.
Another more involved way would be to flash the BMS PIC16F with a custom
firmware, adapted to the desired capacity (as it is, battery capacity is
hardwired in the code).
My guess here is that a FM firmware update to the BMS allowing a maximum
Amp Hours capacity of 5.5 would make the vendor app just work with a Two-X,
but I could be wrong.
My primary method for a Two-X right now is my "Fully Charged Two-X"
option, which is very reliable assuming each time you charge, you charge
until the charger light goes green and you start your ride with 60% or more
battery remaining. Under these circumstances, I keep track of how many amp
hours drawn equals what percentage of the total battery capacity, and I
keep that method going for the user even after the OW things it is at 1%.
If you start a ride with less battery, I fall back to the method of using
the Cell Voltages reported, which is OK for accuracy from 20-40%, pretty
good for accuracy from 10-20%, and reliable for accuracy from 1-10%.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ponewheel/android-ponewheel/issues/76?email_source=notifications&email_token=ADY56OIUFDHLCVC2BE7TF7DQV24MZA5CNFSM4FJ6SV72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFKMR7A#issuecomment-559204604,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADY56ONYZUI3C4TE4UYJB5DQV24MZANCNFSM4FJ6SV7Q
.

I do use amp draw in my calculations, does that come from the BMS? If so, then I would not be able to help @muellergit, because the amp draw is a critical part of smoothing out the output voltage.
It does come from the BMS indeed. The worst considering our issue is that only instantaneous current is transmitted to the controller. And, iirc, to the BLE app.
Also, it's not in mA or any common unit, but a simple integration of 12-bit ADC samplings from the PIC16F. Which explains, I assume, why my version of ponewheel does not report current correctly (understand: in a meaningful unit).

You wouldn't even need pOnewheel then, right? The regular app would mostly work.
Right. In fact, this problem cannot be solved at the app level, afaik. Thus fixing it, whether via a shunt swap or a custom firmware would solve it for all apps. Which isn't a bad thing in itself, tbh.

My testing shows that, on my board, the max stock OW battery only goes to 44% of my total battery on a hot day and 46% on a cold day vs my Two-X capacity. So, it isn't exactly 50%, but close.
These numbers are all nominals. They can vary from one battery pack to another, from on season to another or, in your case from a 1P pack to a 2P one (relationship between current draw and capacity is not linear: if you put 2 packs in parallel, you don't get exactly twice the capacity in practice, you should get more).

My guess here is that a FM firmware update to the BMS allowing a maximum Amp Hours capacity of 5.5 would make the vendor app just work with a Two-X, but I could be wrong.
There is no provision in the BMS firmware for remote update. In other words, the controller cannot upgrade the BMS firmware. So this is not going to happen, ever. :p

Yeah, amp draw is from the BMS so that is why that mode doesn't work on my Zombie.

May I know what happened to your BMS? For what it's worth, i'm fixing them (in case it might help). :)

My testing shows that, on my board, the max stock OW battery only goes to 44% of my total battery on a hot day and 46% on a cold day vs my Two-X capacity. So, it isn't exactly 50%, but close.

These numbers are all nominal. They can vary from one battery pack to another, from one season to another or, in your case, from a 1P pack to a 2P one (relationship between current draw and capacity is not linear: if you put 2 packs in parallel, you don't get exactly twice the capacity in practice, you should get more).

Right, once I have more data, and other Two-X users try the app, then I was thinking of making an application setting which could control for variations in the cell packs used and battery age/charge cycles.  The BMS will do a lot of that for me, it just might mean that 44,45,46 will become 43,44,45 (or something similar).
If you came up with an easy hardware fix, I would consider it.  If you could keep us up-to-date on your progress, that would be great.  Right now, for my board, I can reliably draw it from 100 to 1% on every ride with my most up-to-date code.  I know it isn't perfect, but it is getting the job done for me.  With some external testing and code tweaking, I think it would work well enough for a bunch of other users too.

If you came up with an easy hardware fix, I would consider it. If you could keep us up-to-date on your progress, that would be great.

Sure. Right now i've got a beta firmware for XR BMS. OW+ is a slightly different chip with a slightly different pinout, but i'll get there eventually. ^^

Sorry for a rookie question and notification noise, but how does one get ahold of beta firmware exactly? Through reverse engineering the official app code and then guessing the update service URLs by appending ascending firmware revision numbers until there's a hit for yet-unreleased, but nonetheless at-the-URL-served firmware?
I'm asking this because I'd like to inspect the firmware of the Pint. I have no hardware experience so there's no way I'm going to be able to extract it from the chip's EEPROM and AFAIK there were no OTA updates to the Pint's firmware yet so presumably the update service the app uses shouldn't carry any updates yet? (I'm assuming the original firmware blob is never served by the update service since it comes with the board from the factory and only the OTA revisions make it to the update repository.)

@TomasHubelbauer  There are several micro-controllers on a OW, each with their own firmware. In this thread we're discussing the BMS firmware, which is responsible for battery monitoring and the Two-X capacity issue. From what I understand, you're looking for the controller firmware (the one that gets udpdated OTA), another beast entirely. :-)

@ocornu Thanks for entertaining the question and for the clarification. This is definitely fascinating. I will try to look into this further for a few mods that I have in mind. Thanks again and have a good one 🙂

@biell I was just thinking of a potential new "Zombie" method to estimate battery capacity remaining from voltage alone while moving. I have been riding my Plus without the BMS for about a year now so to estimate my charge remaining I need to stop to check battery voltage in the app (or look at the power button pulses). Since your new Voltage based battery code uses the current amps to compensate for voltage sag, they don't work for my Zombie Plus without the BMS.
What about using a moving peak voltage over the previous minute, but throw out any voltage peak that is more than a certain amount (0.5v perhaps) above the current peak to filter out braking regen voltage spikes? Just wondering your thoughts about it based on what you have done.
Before my BMS failed I used Ponewheel to log data and so I will try to run a few simulations of the method using that data to see how noisy it is and if it can be smoothed enough to be useful. Thanks for any thoughts you have about how well this might work.

We can work on that.  Right now I have major responsibilities at work to support our COVID-19 response.  But, I will help you work on this as time allows.  It will be easy to add, but it will take time to tweak.
Basically, all we need to do is add another option in the drop-down, then make an output choice for that shared preference.  All the variables are there, we will just be using fewer of them to make the guess.  What you want to do is only about an hour of work for revision 1, once you have an idea of how the math should play out.
Maybe also think about rounding to the nearest 5 above 20 to smooth it out, as the fluctuations are quite heavy when there is a lot of battery left.

I need to write a little code to run through my data to see if there is a way to get good smoothness without needing a long moving average. But a long moving average of voltage looks fairly good actually. Especially over an order of 1000 voltage measurements. But my data was sampled once per second so not sure how the smoothing works if sampling was done 10 or 100 times per second.
 There is the TinyWheel project that currently shows the speed and battery level:
https://github.com/FabioBatSilva/TinyWheel

Smoothing out does a pretty good job, and I use that technique with a rolling average.  But, the equation I uses for my Voltage only line is calculated to be used with a layer of smoothing applied from current amp draw.  Without that, the line may need to be adjusted slightly.  We can still make a guess without amp draw, it will just be a little less accurate,  We could substitute something in using speed, but that would probably mess things up when you go downhill.
That device looks pretty cool.  If it supported my XR model, I would totally pick one up for that.

I thought about speed as a correcting factor in place of current but speed isn't well correlated with current since you can be going the same speed while either braking, accelerating, or maintaining speed, and any additional balancing current is completely uncorrelated.
I'm getting useful looking results with Vsmooth = Vsample / 600 + Vsmooth * 599 / 600 which is simple and doesn't need to save any previous voltage samples. Vsmooth just need to initialize Vsmoothprevious to the starting voltage. I can't tell from my once per second data logging I have access to if the sampling rate will make a difference in smoothness using this method, but the Vsmooth curve looks like it tracks the expected battery discharge curve throughout the entire range of the ride. Using 600 as my smoothing "decay" factor was arbitrary also and could be adjusted more or less depending on sampling rate and smoothness needed. It doesn't look like I need to filter out any extreme regen spikes since I am only adding 1/600th of each voltage sample to 599/600th of the previous sample voltage and the positive and negative voltage spikes mostly cancel themselves out. Vsmooth using this formula and my dataset with 1 second sampling rate and 600 decay factor looks like it matches up with what the voltage would be while stopped, but that might just be a lucky coincidence about 600 samples taking 10 minutes and my riding. Not sure if you follow, but I think I have a good starting point to try out while riding and adjust based on results.
Do you know how often the Bluetooth parameters are pulled in Ponewheel?

That is a much larger decay than my decaying average, I divide by 50:
private static final double DECAY_NEW_STEP     = 0.02;
private static final double DECAY_OLD_STEP     = 1.0-DECAY_NEW_STEP;

I can't remember how often the data comes in, but is more like seconds per data element, not data elements per second.


Magnification in auto_sigma functions

Added magnification to auto_sigma_x, auto_sigma_p, auto_sigma_prior, and auto_sigma_y functions.
Tested with demo_fanbeam.py by changing the magnification value around, and look at the calculated sigma_y value and the regularization level of the reconstruction.
sigma_y value doubled when changing magnification factor from 2 to 4, and the reconstructions yield exactly the same NRMSE value and regularization level. Hence the testing results are as expected.

There are a few problems with this modification.
First, I think we haven't settled on the names we are using for the geometries.
Maybe Jordan need to implement that first.
Right now the code is assuming that we will use "fan" but that is probably not true.
Second, I think the order of the arguments in the function call is a problem.
I think all the required arguments must come first.
So the argument weights is in the wrong place.
def auto_sigma_y(sino, geometry = 'parallel', magnification = None, weights, snr_db = 30.0, delta_pixel = None, delta_channel = 1.0):
Do these function even need to take the geometry as an argument?
Maybe they can just take magnification and have it default to 1 for the parallel beam case.


Timeseries with zero median causes Lightkurve to complain about NaN values

Hi,
The following snippet
import numpy as np
import pbjam

N = 1000
timeseries = np.vstack([np.arange(N)*1800., np.random.randn(N)])
timeseries[1] -= np.median(timeseries[1])
timeseries[1] -= np.median(timeseries[1]) # yes, I mean it
s = pbjam.session(ID='0', numax=(80.0, 5.0), dnu=(6.3, 0.2),
                  teff=(5000, 50), bp_rp=(1.0, 0.01), make_plots=True,
                  timeseries=timeseries)
s()

causes this error in lightkurve:
/home/wball/.local/lib/python3.7/site-packages/lightkurve/periodogram.py:744: LightkurveWarning: Input light curve will be normalized.
  LightkurveWarning)
Traceback (most recent call last):
  File "PBjam_bug_3.py", line 14, in <module>
    timeseries=timeseries)
  File "/home/wball/pypi/PBjam/pbjam/jar.py", line 604, in __init__
    lk_to_pg(vardf)
  File "/home/wball/pypi/PBjam/pbjam/jar.py", line 431, in lk_to_pg
    vardf.loc[i, key] = lk_lc.to_periodogram(freq_unit=units.microHertz, normalization='psd').flatten()
  File "/home/wball/.local/lib/python3.7/site-packages/lightkurve/lightcurve.py", line 1230, in to_periodogram
    return LombScarglePeriodogram.from_lightcurve(lc=self, **kwargs)
  File "/home/wball/.local/lib/python3.7/site-packages/lightkurve/periodogram.py", line 760, in from_lightcurve
    raise ValueError('Lightcurve contains NaN values. Use lc.remove_nans()'
ValueError: Lightcurve contains NaN values. Use lc.remove_nans() to remove NaN values from a LightCurve.

This seems to be because lightkurve insists on normalising the light curve by dividing by the median, which presumably leads to non-finite values that lightkurve's error describes as NaNs (even though I'm pretty sure the problem is Infs, not NaNs). I noticed that, when the timeseries is provided as a filename, PBjam handles this (in jar.lc_to_lk()) by adding a small offset but this doesn't seem to apply if the timeseries is provided as an array.

Thanks @warrickball - I think this all comes from the normalisation. We should put an if statement around the LC prep that doesn't normalise if LC.flux.mean() < tol.

This particular aspect of LightKurve is being overhauled at the moment. Should be fixed by requiring PBjam updates to the latest version of LightKurve (in the near future).

This should be fixed now with v1.2.0. We should try it out, I'm not sure if behaviour will be consistent.

Okay so having played around with it, to retain periodogram units of ppm^2/microHz, we need to call:
lc.normalize(unit='ppm').to_periodogram(normalization='psd')
whereas before we simply called:
lc.normalize().to_periodogram(normalization='psd').
I think this should work for zero-centered data as well, but I haven't tried that yet.

Thanks @ojhall94 - we should update the dependency to be lightcurve==1.2.0 and then put in the updated unit in normalize.  @alexlyttle - do you want to have a swing at this one?

I could give that a go, will update lightkurve and this out on my fork.

@alexlyttle - do you know about the dependencies in setup.py?  If no, let's talk tomorrow.

@grd349 I think so, I believe it's just a case of changing lightkurve>=1.0.1 to lightkurve>=1.2.0 in setup.py to make sure that the version of lightkurve installed is greater than or equal to 1.2.0.

If you guys are going to be playing with the dependencies in setup.py, can
you make it pull the dependencies from requirements.txt? Then we only need
to update the dependencies in one place.
It could maybe be done with an easy open('requirements.txt', 'r').read()
call in the setup.py, but I'm not sure.
The requirements.txt is what sphinx/readthedocs uses to make compile
everything for the docs page.
On Thu, 3 Oct 2019 at 11:50, Guy Davies<EMAIL_ADDRESS>wrote:

@alexlyttle https://github.com/alexlyttle - do you know about the
dependencies in setup.py? If no, let's talk tomorrow.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/grd349/PBjam/issues/159?email_source=notifications&email_token=AEJWO324MVSDVIRFUKCSV7TQMXE6RA5CNFSM4ITPFIR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAHZTCY#issuecomment-537893259,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEJWO34B5LKNTLZWA2DMI7DQMXE6RANCNFSM4ITPFIRQ
.

@nielsenmb that makes sense, I can also try that on my fork and see what happens. Will chat with @grd349 tomorrow about it too.

There appears to be a bug in the lightkurve normalize function, so you get the wrong scale of the offset on the flux when using normalize('ppm'). Basically it looks like they've forgotten to subtract 1 when converting from flux to ppm, so you wind up with a mean of 1e6, rather than 0 after converting.
It might still work for our purposes though, since to_periodogram should handle offsets in the input time series.
Just so you're aware

@nielsenmb thanks for the heads-up!

@nielsenmb I modified the normalize() function in Lightkurve so that it takes away 1 from the flux before converting to percent, ppt and ppm. I intend to create a pull request/issue on lightkurve about this, but want to mention some of the side-effects of the change first.
For example, I noticed flattening the light curve before/after normalisation massively changes the appearance of the light curve. This could be because of the behaviour of lightcurve.flatten() near zero. See the attached pdf for plots (using the example star in PBjam docs):
lightkurve-test.pdf
This might be more appropriate in a separate Lightkurve issue but I thought it would be useful here too.

I tried running the script in the first comment of this thread and it throws up an error when trying to initialise session. @warrickball, do you know which version of PBjam you ran this on? I was looking back to see if there was a change in the code somewhere that is causing this, but it is pretty difficult to pinpoint so I will leave it here for now.
Here is the error which arises after running the script, using PBjam version 0.1.6 on MacOS 10.15:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2896             try:
-> 2897                 return self._engine.get_loc(key)
   2898             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'timeseries'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/internals/managers.py in set(self, item, value)
   1068         try:
-> 1069             loc = self.items.get_loc(item)
   1070         except KeyError:

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2898             except KeyError:
-> 2899                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2900         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'timeseries'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-2-ebe582da4175> in <module>
      1 s = pb.session(ID='0', numax=(80.0, 5.0), dnu=(6.3, 0.2),
      2                teff=(5000, 50), bp_rp=(1.0, 0.01),
----> 3                timeseries=timeseries)

~/Projects/PBjam/pbjam/session.py in __init__(self, ID, numax, dnu, teff, bp_rp, timeseries, psd, dictlike, use_cached, cadence, campaign, sector, month, quarter, path, download_dir)
    620                                         campaign=campaign, sector=sector,
    621                                         month=month, quarter=quarter)
--> 622             format_col(vardf, timeseries, 'timeseries')
    623             format_col(vardf, psd, 'psd')
    624

~/Projects/PBjam/pbjam/session.py in format_col(vardf, col, key)
    357         x = np.array(col[0, :], dtype=float)
    358         y = np.array(col[1, :], dtype=float)
--> 359         vardf[key] = np.array([arr_to_lk(x, y, vardf['ID'][0], key)])
    360
    361     # If dim = 3, it's a list of arrays or tuples

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/frame.py in __setitem__(self, key, value)
   3470         else:
   3471             # set column
-> 3472             self._set_item(key, value)
   3473
   3474     def _setitem_slice(self, key, value):

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/frame.py in _set_item(self, key, value)
   3548         self._ensure_valid_index(value)
   3549         value = self._sanitize_column(key, value)
-> 3550         NDFrame._set_item(self, key, value)
   3551
   3552         # check if we are modifying a copy

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/generic.py in _set_item(self, key, value)
   3379
   3380     def _set_item(self, key, value):
-> 3381         self._data.set(key, value)
   3382         self._clear_item_cache()
   3383

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/internals/managers.py in set(self, item, value)
   1070         except KeyError:
   1071             # This item wasn't present, just insert at end
-> 1072             self.insert(len(self.items), item, value)
   1073             return
   1074

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/internals/managers.py in insert(self, loc, item, value, allow_duplicates)
   1179         new_axis = self.items.insert(loc, item)
   1180
-> 1181         block = make_block(values=value, ndim=self.ndim, placement=slice(loc, loc + 1))
   1182
   1183         for blkno, count in _fast_count_smallints(self._blknos[loc:]):

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/internals/blocks.py in make_block(values, placement, klass, ndim, dtype, fastpath)
   3265         values = DatetimeArray._simple_new(values, dtype=dtype)
   3266
-> 3267     return klass(values, ndim=ndim, placement=placement)
   3268
   3269

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/internals/blocks.py in __init__(self, values, placement, ndim)
   2773             values = np.array(values, dtype=object)
   2774
-> 2775         super().__init__(values, ndim=ndim, placement=placement)
   2776
   2777     @property

~/.virtualenvs/pbjam/lib/python3.7/site-packages/pandas-0.25.1-py3.7-macosx-10.14-x86_64.egg/pandas/core/internals/blocks.py in __init__(self, values, placement, ndim)
    126             raise ValueError(
    127                 "Wrong number of items passed {val}, placement implies "
--> 128                 "{mgr}".format(val=len(self.values), mgr=len(self.mgr_locs))
    129             )
    130

ValueError: Wrong number of items passed 1000, placement implies 1```

Getting the same error as Alex, investigating...

Seems like the offending line is in session.py line
359         vardf[key] = np.array([arr_to_lk(x, y, vardf['ID'][0], key)])
For some obscure reason when the output from arr_to_lk is put in an array, it messes up something with creating a new column in a PandasDataframe.
Replacing it with
vardf[key] = [arr_to_lk(x, y, vardf['ID'][0], key)]
seems to have solved this for me...

That's odd, but if that fixes it then it should be fine. I'll give that a go and add it to the pull request if it works.

I don't think I've updated anything since opening this issue and I still get the same error. I seem to have PBjam version 0.1.0:
$ pip3 show pbjam
Name: pbjam
Version: 0.1.0
...

I won't try upgrading yet in case it makes it hard to reproduce the error.

Cheers @warrickball. In the latest version of PBjam on git and with lightkurve v1.2.0, after adding to my fork the fix suggested by @nielsenmb, I don't get the lightkurve error you had, or the pandas KeyError I quoted.
Now I just get this warning upon initialising session:
UserWarning: Input numax is greater than Nyquist frequeny for 0

but otherwise it initialises fine. Then, when I call session it complains that the prior is very large and may be slow - presumably because the custom timeseries is random so it's going to struggle?

Yup, the numax warning is probably cause you are using a fake time series.
You can try using a real timeseries.
The warning about the slow KDE occurs when the prior data it's being
trained on contains more than 1000 targets. This probably occurs because of
some combination of large errors on numax and dnu and that you are in a
particularly dense part of the prior sample.
On Thu, 10 Oct 2019 at 13:52, Alex Lyttle<EMAIL_ADDRESS>wrote:

Cheers @warrickball https://github.com/warrickball. In the latest
version of PBjam on git and with lightkurve v1.2.0, after adding to my fork
the fix
https://github.com/grd349/PBjam/issues/159#issuecomment-540424072
suggested by @nielsenmb https://github.com/nielsenmb, I don't get the
lightkurve error you had, or the pandas KeyError I quoted
https://github.com/grd349/PBjam/issues/159#issuecomment-540093168.
Now I just get this warning upon initialising session:
UserWarning: Input numax is greater than Nyquist frequeny for 0
but otherwise it initialises fine. Then, when I call session it complains
that the prior is very large and may be slow - presumably because the
custom timeseries is random so it's going to struggle?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grd349/PBjam/issues/159?email_source=notifications&email_token=AEJWO345DVGQBKDCTJ4WBMTQN33PVA5CNFSM4ITPFIR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA3ULJA#issuecomment-540493220,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEJWO37B4JZS2DMAEMJYC6DQN33PVANCNFSM4ITPFIRQ
.

It works fine using a real time series, no warnings and no issues after implementing your suggested fix @nielsenmb.

Cool, feel free to commit the changes to your pull request, and I'll merge
it into the master branch
On Thu, 10 Oct 2019 at 14:30, Alex Lyttle<EMAIL_ADDRESS>wrote:

It works fine using a real time series, no warnings and no issues after
implementing your suggested fix @nielsenmb https://github.com/nielsenmb.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grd349/PBjam/issues/159?email_source=notifications&email_token=AEJWO365FVTL5MKU6ZYED7DQN3727A5CNFSM4ITPFIR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA3XXSQ#issuecomment-540507082,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEJWO3ZWCFNGINNU4ZEJI7DQN3727ANCNFSM4ITPFIRQ
.

I upgraded yesterday afternoon so now have version 0.1.6 (and lightkurve 1.2.0). The same script seems to run to completion with the following output, which may or may not be worrying:
/home/wball/pypi/PBjam/pbjam/session.py:645: UserWarning: Input numax is greater than Nyquist frequeny for 0
  warnings.warn("Input numax is greater than Nyquist frequeny for %s" % (st.ID))
/home/wball/pypi/PBjam/pbjam/priors.py:66: UserWarning: You have lots data points in your prior - estimating the KDE band width will be slow!
  warnings.warn('You have lots data points in your prior - estimating' +
 72%|████████████████████████████████████████████▏                | 14496/20000 [06:08<02:15, 40.61it/s]Converged after 14500 iterations.
 72%|████████████████████████████████████████████▏                | 14500/20000 [06:08<02:19, 39.32it/s]
100%|█████████████████████████████████████████████████████████████| 20000/20000 [11:19<00:00, 29.42it/s]
Star 0 produced an exception of type ValueError occurred. Arguments:
('min() arg is an empty sequence',)

I've run it twice and got the same result so it probably isn't random.


fix: broken link of README.md

What type of PR is this?
/kind documentation

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?

Additional documentation e.g., design docs, usage docs, etc.:

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.006%) to 54.369%

Files with Coverage Reduction
New Missed Lines
%

pkg/engine/operation/watch.go
1
82.76%

Totals

Change from base Build<PHONE_NUMBER>:
-0.006%

Covered Lines:
9109

Relevant Lines:
16754

💛  - Coveralls


Fixes #32 - Cast the null to shared(Driver) to appease the cas template.

ping @jondegenhardt I think this will have it build again. Can you test possibly before I merge?

I'm pretty busy today, I might be able to get to it tonight though.

OK, looks like all the things that were failing on CI are now green. So I'm just going to pull the lever and maybe your CI will go green as well.

Last time you updated the version number in the meson build file, but I don't see a change to that file. Did you miss this?

Ugh, yes. That was a problem I was concerned with (forgetting to update this build number), but it apparently is expected practice. Will update and move the tag.

For my own repos I have a "steps to do a release" notes file. Otherwise I'd forget. Little stuff like signing the commit tag.

@jondegenhardt updated meson build, and moved the tag.

Build succeeded on all compiler versions except for DMD v2.089.0. See https://travis-ci.com/jondegenhardt/dcat-perf/builds/149768223. The CI has 17 different DMD/LDC/Linux/OSX combos, so its got fair coverage.

However, that my CI doesn't explicitly test DMD 2.090.0. It does test latest, which is DMD 2.090.1. That release passed.

That was a regression, 2.089.1 works.
https://github.com/dlang/druntime/pull/2853

Excellent, thanks. I'll remove 2.089.0 from my CI tests.


404 Exception if no endpoints are configured

Hey,
if no endpoints are configured in config.yml, a 404 Exception is thrown, stating that the request handler cannot be accessed. Reason: Not found. It would be more helpful if the programmer would be notified that no endpoints are defined (or that the config is missing). Would make the implementation more robust since it takes some time to find the error source.
Thanks in advance!

I agree it'd be nice, but it's not really high priority. If someone contributes it though I'm all for it :)

Hey,
just saw that you switch to a default endpoint if no explicit endpoint is specified by the user. Replacing that by an exception in exceptionRequest() would solve the issue but I'm not sure if this is your intended behavior? If so I could create a pull request. Do you have reasons for a default endpoint instead of a thrown error?

Oh I see, well I guess because most people just run it on localhost with default config, that usually works fine. Throwing without config will cause problems for people that use the default right now so I don't think we can really change that.

Alright then we leave it like that! Thanks anyway.


Allow overriding the link search path with <TARGET>_MAGIC_DIR

In a way similar to openssl-sys this allows the user to pass a MAGIC_DIR environment variable that tells rustc where to find libmagic.so/libmagic.a.  For finer grained control, the user can pass an environment variable like x86_64_UNKNOWN_LINUX_MUSL_MAGIC_DIR to only set the custom search path for a specific target.
Static linking is also now possible, detected either by only a libmagic.a found in the (provided) search path, or by explicitly passing MAGIC_STATIC=true (or <TARGET>_MAGIC_STATIC=true).  Similarly MAGIC_STATIC=false can be passed to choose to link dynamically.  If neither is passed, but both libraries are available, the build.rs bails out and asks the user to choose an implementation, an alternative would be to pick one (openssl-sys picks "shared library" in this case).
Tested with a local, static-only, musl-gcc-compiled libmagic:
$ X86_64_UNKNOWN_LINUX_MUSL_MAGIC_STATIC=true X86_64_UNKNOWN_LINUX_MUSL_MAGIC_DIR=../libmagic/src/.libs cargo c -vv --target x86_64-unknown-linux-musl
...
     Running `/home/andy/src/rust-magic-sys/target/debug/build/magic-sys-3a925b5dcd0ba125/build-script-build`
[magic-sys 0.2.0] cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_MUSL_MAGIC_DIR
[magic-sys 0.2.0] cargo:rustc-link-search=native=../libmagic/src/.libs
[magic-sys 0.2.0] cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_MUSL_MAGIC_STATIC
[magic-sys 0.2.0] cargo:rustc-link-lib=static=magic
     Running `CARGO=/home/andy/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo CARGO_MANIFEST_DIR=/home/andy/src/rust-magic-sys CARGO_PKG_AUTHORS='robo9k<EMAIL_ADDRESS>CARGO_PKG_DESCRIPTION='Declarations for `libmagic`' CARGO_PKG_HOMEPAGE='https://github.com/robo9k/rust-magic-sys' CARGO_PKG_NAME=magic-sys CARGO_PKG_REPOSITORY='https://github.com/robo9k/rust-magic-sys.git' CARGO_PKG_VERSION=0.2.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=2 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= LD_LIBRARY_PATH='/home/andy/src/rust-magic-sys/target/debug/deps:/home/andy/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib:/home/andy/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' OUT_DIR=/home/andy/src/rust-magic-sys/target/x86_64-unknown-linux-musl/debug/build/magic-sys-e078dc43334cb961/out sccache-msw rustc --crate-name magic_sys src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata -Cembed-bitcode=no -C debuginfo=2 -C metadata=2a7a520a13672ee4 -C extra-filename=-2a7a520a13672ee4 --out-dir /home/andy/src/rust-magic-sys/target/x86_64-unknown-linux-musl/debug/deps --target x86_64-unknown-linux-musl -C incremental=/home/andy/src/rust-magic-sys/target/x86_64-unknown-linux-musl/debug/incremental -L dependency=/home/andy/src/rust-magic-sys/target/x86_64-unknown-linux-musl/debug/deps -L dependency=/home/andy/src/rust-magic-sys/target/debug/deps --extern libc=/home/andy/src/rust-magic-sys/target/x86_64-unknown-linux-musl/debug/deps/liblibc-df81d5c25afc76a4.rmeta -L native=../libmagic/src/.libs -l static=magic`
    Finished dev [unoptimized + debuginfo] target(s) in 0.92s

If neither environment variable is set, the behaviour is completely unchanged (we link dynamically with the system-wide library).

I'm using rust-magic-sys through rust-magic, but I don't think I need any changes to rust-magic to be able to use this (assuming this is released in a version of rust-magic-sys that's semver-compatible with the previous one, which I think should be possible since the fallback  behaviour is unchanged?).
The only thing I'm not sure about is where the documentation for this should live, maybe leave the docs in this repo's README and link to them from rust-magic's README with something like:

By default rust-magic will search your system library paths for a version of libmagic.so, if you're cross-compiling, or need more control over which library is selected, see here.

@bossmc I've updated this repo's README and reference it in rust-magic's README. I've also published a new 0.2.1 version of rust-magic-sys, which should be semver compatible with the 0.2.0 from rust-magic's requirements.
CI build of rust-magic is failing, but that's unrellated. Please give it a try

Thanks, that's working for me!
$ X86_64_UNKNOWN_LINUX_MUSL_MAGIC_STATIC=true X86_64_UNKNOWN_LINUX_MUSL_MAGIC_DIR=$PWD/../libmagic/src/.libs cargo b --target x86_64-unknown-linux-musl
   Compiling magic-sys v0.2.1
   Compiling magic v0.12.2
   Compiling magic-test v0.1.0 (/home/andy/src/magic-test)
    Finished dev [unoptimized + debuginfo] target(s) in 0.78s
$ ldd target/x86_64-unknown-linux-musl/debug/magic-test
	not a dynamic executable


IQN

Description

Context

[ ] I have raised an issue to propose this change (required)

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation (update in the documentation)

Checklist:

[x] I've read the CONTRIBUTION guide (required)
[ ] The functionality/performance matches that of the source (required for new training algorithms or training-related features).
[ ] I have updated the tests accordingly (required for a bug fix or a new feature).
[ ] I have included an example of using the feature (required for new features).
[ ] I have included baseline results (required for new training algorithms or training-related features).
[ ] I have updated the documentation accordingly.
[ ] I have updated the changelog accordingly (required).
[x] I have reformatted the code using make format (required)
[ ] I have checked the codestyle using make check-codestyle and make lint (required)
[ ] I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

Results comparison
Current implementation (6 seeds)

Reference from https://github.com/toshikwa/fqf-iqn-qrdqn.pytorch (1 seed, same parameters):

@qgallouedec Thank you for adding this. I wanted to report that for me it works well and I was able to adapt it to implement the paper Self-Imitation Advantage Learning
. I'm not sure how useful it is for you but I'm happy to share my modifications to add SAIL-IQN to your IQN implementation (I don't have the resources right now to submit this as a separate PR):
New replay buffer to store discounted returns (G):
import warnings
import itertools
from typing import Generator, Optional, Union, NamedTuple, List, Dict, Any
import numpy as np
import torch as th
from stable_baselines3.common.type_aliases import ReplayBufferSamples, RolloutBufferSamples
from stable_baselines3.common.buffers import ReplayBuffer
from stable_baselines3.common.vec_env import VecNormalize
from gymnasium import spaces

PLACEHOLDER_RETURN_VALUE = np.finfo(np.float32).min

class SAILReplayBufferSamples(NamedTuple):
    observations: th.Tensor
    actions: th.Tensor
    next_observations: th.Tensor
    dones: th.Tensor
    rewards: th.Tensor
    returns: th.Tensor

class SAILReplayBuffer(ReplayBuffer):
    def __init__(
        self,
        buffer_size: int,
        observation_space: spaces.Space,
        action_space: spaces.Space,
        device: Union[th.device, str] = "cpu",
        n_envs: int = 1,
        optimize_memory_usage: bool = False,
        gamma: float = 0.99
    ):
        super().__init__(buffer_size, observation_space, action_space, device, n_envs, optimize_memory_usage)
        ## TODO: Haven't looked at supporting optimize_memory_usage true yet
        # assert optimize_memory_usage == False, 'optimize_memory_usage does not work with SAIL currently'
        self.gamma = gamma
        self.returns = np.zeros((self.buffer_size, self.n_envs), dtype=np.float32)
        # For each env store where the episode starts (0 for all envs at the beginning)
        # but will vary as each episode can end at a different point
        self.episode_start_indices = np.zeros(self.n_envs, dtype=np.int32)

    def update_episodic_return(self, completed_env_indices: np.ndarray, episode_end_idx: int):
        # completed_env_indices - indices of envs with completed episode
        # For all episodes that have ended episode_end_pos will contain the end position though
        # it may be infrequent for multiple episodes to end at the same position
        for env_idx in completed_env_indices:
            # episode_start_idx can be > episode_end_idx due to buffer wrap-around
            episode_start_idx = self.episode_start_indices[env_idx]
            G = 0
            x = 0
            i = episode_end_idx # index used to calculate discounted return
            if episode_start_idx < episode_end_idx:
                max_episode_steps = episode_end_idx - episode_start_idx
            else:
                # This won't be accurate if we've wrapped around more than once but we should somewhere require
                # max_episode_steps to be less than buffer size to prevent that from happening.
                max_episode_steps = self.buffer_size - episode_end_idx + episode_start_idx
            while x <= max_episode_steps:
                G = self.rewards[i, env_idx] + self.gamma * G
                self.returns[i, env_idx] = G
                i = (i - 1) % self.buffer_size
                x += 1
                pass

        pass
    def add(
        self,
        obs: np.ndarray,
        next_obs: np.ndarray,
        action: np.ndarray,
        reward: np.ndarray,
        done: np.ndarray,
        infos: List[Dict[str, Any]],
    ) -> None:
        # we want position before it gets updated
        pos = self.pos
        super().add(obs=obs, next_obs=next_obs, action=action, reward=reward, done=done, infos=infos)
        self.returns[pos] = np.repeat(PLACEHOLDER_RETURN_VALUE, repeats=self.n_envs)
        done[1] = True
        pos = 9
        self.rewards[0:10, 1] = 1
        if np.any(done):
            # Only use dones that are not due to timeouts
            true_dones = done * (1 - self.timeouts[pos])
            if np.any(true_dones):
                self.update_episodic_return(np.flatnonzero(true_dones), pos)
            # Update episode start indices (whether due to timeout or not) to the current start index
            # of the next episode (self.pos)
            np.put_along_axis(self.episode_start_indices, np.flatnonzero(done), self.pos, axis=0)
            pass
        return

    def sample(self, batch_size: int, env: Optional[VecNormalize] = None) -> SAILReplayBufferSamples:
        # noinspection PyTypeChecker
        return super().sample(batch_size=batch_size, env=env)

    def _get_samples(self, batch_inds: np.ndarray, env: Optional[VecNormalize] = None) -> SAILReplayBufferSamples:
        # Sample randomly the env idx
        env_indices = np.random.randint(0, high=self.n_envs, size=(len(batch_inds),))

        if self.optimize_memory_usage:
            next_obs = self._normalize_obs(self.observations[(batch_inds + 1) % self.buffer_size, env_indices, :], env)
        else:
            next_obs = self._normalize_obs(self.next_observations[batch_inds, env_indices, :], env)

        data = (
            self._normalize_obs(self.observations[batch_inds, env_indices, :], env),
            self.actions[batch_inds, env_indices, :],
            next_obs,
            # Only use dones that are not due to timeouts
            # deactivated by default (timeouts is initialized as an array of False)
            (self.dones[batch_inds, env_indices] * (1 - self.timeouts[batch_inds, env_indices])).reshape(-1, 1),
            self._normalize_reward(self.rewards[batch_inds, env_indices].reshape(-1, 1), env),
            self.returns[batch_inds, env_indices]
        )
        return SAILReplayBufferSamples(*tuple(map(self.to_torch, data)))

and updated training loop:
        # Sample replay buffer
        replay_data = self.replay_buffer.sample(batch_size, env=self._vec_normalize_env)
        with th.no_grad():
            # BEGIN - SAIL addition.
            rewards = replay_data.rewards
            # Ref to https://github.com/google-research/google-research/blob/master/sail_rl/agents/sail_iqn.py
            # Calculates **current state** action-values.
            # Shape: batch_size x n_quantiles x num_actions.
            replay_target_net_outputs = self.quantile_net_target(replay_data.observations, self.n_quantiles)

            # Shape: batch_size x num_actions
            replay_target_q_values = replay_target_net_outputs.mean(dim=1)#, keepdim=True)

            replay_action_one_hot = th.nn.functional.one_hot(replay_data.actions.squeeze(-1), self.action_space.n).type(th.float32)
            replay_target_q = th.max(replay_target_q_values, dim=1).values
            replay_target_q_al = th.sum(replay_action_one_hot * replay_target_q_values, dim=1)
            comp_value = th.max(replay_target_q_al, replay_data.returns)

            if self.clip > 0.:
                sil_bonus = self.alpha * th.clamp(comp_value - replay_target_q, min=-self.clip, max=self.clip)
            else:
                sil_bonus = self.alpha * (comp_value - replay_target_q)

            rewards = rewards + sil_bonus.unsqueeze(-1)
            # END - SAIL addition

            # Compute the quantiles of next observation
            next_quantiles = self.quantile_net_target(replay_data.next_observations, self.n_quantiles)

            # Shape of next_quantiles:
            # batch_size x n_quantiles x num_actions.
            # e.g. if num_actions is 2, it might look something like this:
            # Vals for Quantile .2  Vals for Quantile .4  Vals for Quantile .6
            #    [[0.1, 0.5],         [0.15, -0.3],          [0.15, -0.2]]
            # Q-values = [(0.1 + 0.15 + 0.15)/3, (0.5 + 0.15 + -0.2)/3].

            # Compute the greedy actions which maximize the next Q values
            next_greedy_actions = next_quantiles.mean(dim=1, keepdim=True).argmax(dim=2, keepdim=True)

            # Make "num_tau_prime_samples" copies of actions, and reshape to (batch_size, num_tau_prime_samples, 1)
            next_greedy_actions = next_greedy_actions.expand(batch_size, self.num_tau_prime_samples, 1)

            # Compute the quantiles of next observation, but with another number of tau samples
            next_quantiles = self.quantile_net_target(replay_data.next_observations, self.num_tau_prime_samples)

            # Follow greedy policy: use the one with the highest Q values
            next_quantiles = next_quantiles.gather(dim=2, index=next_greedy_actions).squeeze(dim=2)

            # 1-step TD target
            target_quantiles = rewards + (1 - replay_data.dones) * self.gamma * next_quantiles

        # Get current quantile estimates
        current_quantiles = self.quantile_net(replay_data.observations, self.num_tau_samples)

        # Make "num_tau_samples" copies of actions, and reshape to (batch_size, num_tau_samples, 1).
        actions = replay_data.actions[..., None].long().expand(batch_size, self.num_tau_samples, 1)

        # Retrieve the quantiles for the actions from the replay buffer
        current_quantiles = th.gather(current_quantiles, dim=2, index=actions).squeeze(dim=2)

        # Compute Quantile Huber loss, summing over a quantile dimension as in the paper.
        loss = quantile_huber_loss(current_quantiles, target_quantiles, sum_over_quantiles=True)
        return loss

The extra parameters alpha and clip are defaulted to 0.9 and 1.0.
I found immediately that SAIL-IQN performs nicely on sparse rewards so am quite happy with my initial results but by no means has my testing been thorough.

Thanks for your feedback @emrul! This PR is still draft because I can't replicate exactly the results of the paper for Qbert. I don't know if it's a hyperparameter problem or something else, I'm still looking.
I think SIL (and probably maybe SAIL) would fit in SB3-contrib. However, it would be best to discuss it in a dedicated issue. I'll open it right away.

Thanks @qgallouedec - I didn't know there's a reproduction issue, I will look into this also - I compared your implementation with the Dopamine one and the Medipexel/pytorch port of that and it looked quite different. I will dig in to see where they differ and feedback if I find anything to assist.

@qgallouedec So far, I notice the output of the Cosine Embedding Net is not passed through an activation function whereas both the Dopamine implementation and the Medipixel implementation appear to pass through a ReLU before multiplying by observation.


PushIOManager.getPreference uses incorrect "value" parameter for iOS

Documentation suggests accepts PushIOManager.getPreference(options: {key: string}) however on iOS this does not work.
Looks like the iOS version accepts "value" as a parameter not "key"
plugins/capacitor-pushiomanager/ios/Plugin/PushIOManagerPlugin.swift
@objc func getPreference(_ call: CAPPluginCall) {

        if let value = call.getString("value"){
            if let preference = pushIOHandler.getPreference(key: value),let json = preference.dictionaryFromPreference()?.json(){
                    call.resolve(["success": json])
            } else {
                call.resolve(["failure":"preference is not available"])
            }

        } else {
            call.resolve(["failure":"provide preference key"])
        }

    }

Thanks @ichaelpham
iOS should also accept key.
We will change this.

PushIOManager.getPreference({ key: "sampleKey" }) nows works for iOS as well.


Ninja

Implementation of #425
On non-Windows Ninja is used by default. On Windows it can be used with flag --ninja
I have successfully used it by default in npm and builded popular modules on Linux64, Linux32, Windows.
Ninja is setting up in install command. For Windows we download a prebuild binary, for others we build it from sources.

+1
good if that solves problems with modules installation on Windows

Closing because of no activity here for years.


Missing SPHERIC_GTO_PARITY_ODD entries for l-functions and higher

I'm trying to run some calculations with an octuple-zeta basis set, which contains functions with angular momentum up to and including 8 (i.e. l-functions, if we follow the sometimes-convention of skipping j and enumerating as s, p, d, f, g, h, i, k, l). Point-group symmetry would be highly desirable for these calculations, but I'm seeing a crash whenever I try to enable it.
The issue seems to be that there is no appropriate entry in pyscf.symm.params.SPHERIC_GTO_PARITY_ODD, which only supports up to what I know as k-functions (but labeled j in the comments, so I guess there is some disagreement on naming conventions :wink: ).
If someone has time, would it be possible to add these entries for functions with angular momentum 9, and possibly 10 and/or 11? Thanks very much in advance!

The table should be extended to L=15 which I believe is the maximum order supported in PySCF at the moment. I would imagine that the array could be generated automatically...? The array appears to just document the partial inversion symmetry of the solid harmonics.

The connection between the (l,m) indexing and the cartesian components has been discussed by Schlegel and Frisch in Int. J. Quantum Chem. 54, 83 (1995) and the equations are also shown in in https://en.wikipedia.org/wiki/Solid_harmonics#Real_form. It should be quite straightforward to pick out the parity with respect to x, y, and z from these equations.

Actually, while it's under discussion, it looks like pyscf.symm.basis._SO3_SYMB2ID needs to be updated as well.

@glowthrower is everything working now? I just created an issue in the qcint repository because I'm still getting errors with an 8z basis set for the O atom. Are you using qcint or libcint?


Fix #1496: Allow partial paths in jsDependencies.

This comes with an important caveat: all jsDependencies in a dependency graph must agree on exactly which subpath to use to refer to a given .js file within its jar. But at least, it is an improvement over the status quo, where disambiguation is completely impossible.

@gzm0 @fxmouthuy What do you think?

Yes, this would allow us to solve our problem.

To be honest, I am very reluctant about changing something in this mechanism, until we have a story how we do this properly. IMO we are at a point where we need to improve the webjars themselves to store dependency information.
We could have the sbt plugin read the info, and additionally allow a mechanism to inject info, if some webjars are still missing the info.
The initial approach taken was clear not to scale to any real life project, but like this it is even worse it seems as there are now different, incompatible ways to refer to the same JS file.

This is clearly a patch on the hack. But we're not anywhere near changing the way WebJars are made, because that's out of our reach, unfortunately.
This commit clearly trades

Several incompatible IDs for the same file -- which requires people to be careful -- vs
No possible ID for some files -- which makes it impossible to refer to some files, no matter how careful you are.

It seems to me that 2. is the lesser of the two evils.

There is yet another "patch" alternative: consider "full" relative paths as the IDs, for the purpose of deduplication and dependencies, but allow the possibility to refer to files with any (right) subpath. If a subpath can refer to two IDs in a given classpath, fail the resolution. In other words, a subpath is just a short notation for the (supposedly) unique ID of which it is a suffix.
So, for example, for Jasmine. In my jsDependencies, both after / and after dependsOn, I could write (independently) jasmine.js, 1.3.1/jasmine.js, etc. But all of these these suffixes are just short-hand for the full relative path META-INF/resources/webjars/jasmine/1.3.1/jasmine.js. I can have one jsDependencies mentioning jasmine.js, one dependsOn "1.3.1/jasmine.js", and another jsDependencies (through transitive dependencies, for example) mentioning jasmine/1.3.1/jasmine.js. All of them would refer to the one jasmine.js on the classpath.
This would bring a bit more sanity, because it would not have the "incompatible ways to refer to the same JS file" problem. In fact, there are several ways, but they are compatible.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://scala-webapps.epfl.ch/jenkins/job/scalajs-pr/1448/
https://scala-webapps.epfl.ch/jenkins/job/scalajs-matrix-build/1853/
Test PASSed.

I must say I very much prefer the second approach. Although it would be nice to anchor the top path at jasmine/1.3.1 (without the webjar specific fuss)....
But I assume this is hard to impossible to implement reasonably. So maybe anchoring at META-INF is a reasonable compromise.

It doesn't matter where we anchor, since no one would ever write the full path anyway, and it won't be displayed. Only used internally. If we decide to anchor at a non-top-level position, what becomes of jars that do not start with the weird WebJar thing?

Yeah. I know. It's true that writing jasmine/1.3.1/jasmine.js is actually pretty reasonable and the fact that it is just a prefix is just a nasty implementation detail.

And this is even compatible with existing build files out there :)

Superseded by #1510.


Issue creating upload session on a folder

I am trying to create an upload session to upload a file in a folder in Sharepoint. I am receiving the following error: Cannot create an upload session on a folder.
I understand that an upload session cannot be created on a folder, however I am not sure how I can create one.
This is what I am currently doing:
uploadSession, err := graphClient.
		Drives().ByDriveId(driveID).
		Items().ByDriveItemId(folderID).
		CreateUploadSession().
		Post(ctx, uploadSessionRequestBody, nil)

I noticed that the msgraph-sdk-net has a method ItemWithPath which is not available in the Go version (https://github.com/microsoftgraph/msgraph-sdk-dotnet/issues/1353)
var uploadSession = await graphClient.Sites["site-id"].Drive.Items["folderId"].ItemWithPath("file.txt")
                            .CreateUploadSession(uploadProps)
                            .Request()
                            .PostAsync();

Any ideas on how I can do this using msgraph-sdk-go?

Hi @MilitsaB, thanks for tryong out the SDK. Are you able to test this example  i.e concatenate the folderID + itemPath + ":"

uploadSession, _ := graphClient.Drives().
    ByDriveId(*myDrive.GetId()).
    Items().
    ByDriveItemId(folderID + "/"+"file.txt"+":").
    CreateUploadSession().
    Post(context.Background(), uploadSessionRequestBody, nil)


[WIP][NF] Auto-encoders - streamlines - FINTA

Description
Implementation of FINTA using dwi_ml
Testing data and script
None
Have you

[ ] Added a description of the content of this PR above
[ ] Followed proper commit message formatting
[ ] Added data and a script to test this PR
[ ] Made sure that PEP8 issues are resolved
[ ] Tested the code yourself right before pushing
[ ] Added unit tests to test the code you implemented

People this PR concerns
@EmmaRenauld

Hello @arnaudbore, Thank you for submitting the Pull Request !

In the file dwi_ml/data/hdf5/hdf5_creation.py:

Line 747:80: E501 line too long (80 > 79 characters)
Line 748:80: E501 line too long (84 > 79 characters)

In the file dwi_ml/models/projects/ae_models.py:

Line 178:80: E501 line too long (86 > 79 characters)
Line 179:80: E501 line too long (82 > 79 characters)

In the file dwi_ml/training/batch_loaders.py:

Line 197:80: E501 line too long (82 > 79 characters)

In the file scripts_python/ae_train_model.py:

Line 108:80: E501 line too long (83 > 79 characters)

Do see the PEP 8 -- Style Guide for Python Code


Problems with markdown for yaml

File: get-started/08_using_compose.md

The doc page includes

services: app: image: node:12-alpine ```

and many other problems.

The doc page includes

services: app: image: node:12-alpine ```

and many other problems.


Type inference not correct on Loops with Break

Hello!

The type of x should be usize, which Rustc sees and allows to compile without error.
Not the biggest deal, but just thought I'd add an issue

Thanks guys :)


feat(go/adbc/driver/bigquery): Add schema to reader for BigQuery

Fixes: #2049

@zeroshade Is there anything further I can do to help get this PR in?

@zeroshade Is there anything further I need to do to help get this PR in?


Update clr-integration-programming-model-restrictions.md

We believe that publisher policy is not supported based on the following RFC. Can please describe in the documentation that publisher policy is not supported?
RFC 263461: RFC: ALTER ASSEMBLY related questions - Boards (microsoft.com)

@rwestMSFT
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Investigating. Thanks for your contribution!

@katuyaito This change will be incorporated into an internal PR. When that PR is merged, this PR will automatically close. Thank you again for your contribution.


Migrate to npm

Addressing #164
The only issue was with eth-pm where we use @material-ui which has different React version comparing to the root package.json. To fix it I had to apply legacy-peer-deps policy which makes install phase for this example longer.

Nice!


[Haxe4] On haxe4 traces spams Results panel

On haxe4 traces shown also in Results panel grouped by classes
trace("=================");
Haxe4: "src/Main.hx:20: ================="
Haxe3: "Main.hx:20: ================="

This is the new way of how Haxe logs stuff, so it's not FlashDevelop / HaxeDevelop related.


Unstable tests that have always been failing

These are some tests that have not had a successful run in the past week. What should we do about them?

[ ] Object Explorer integration suite Standalone database context menu test @UNSTABLE@
[ ] Dacpac integration test suite Import and export bacpac @UNSTABLE@

#8799


[Controller] Provide highlight of controllers on interactions (touch/grab/use)

It's not possible to change the colour of the controller meshes as the dark textures of the HTC controllers don't lend themselves well to tints.
However, it may be possible to use a different shader and provide an outline colour on the controller models (like as seen in The Lab)

Attempted to use an outline shader on the controller model, but the model shows outlines around all of the sub components and looks quite messy.

I spent some time messing with the 2-camera approach described here:
http://blog.tammearu.eu/index.php/2015/10/09/unity-outline-glow-2-cameras-edge-detection-and-single-color/
It renders proper outlines for objects including the controller models but I could not get the overlay camera to match the FOV of the SteamVR camera or render in the headset. My guess is this is due to how the SteamVR_Camera component submits frames to the compositor which seems to preclude multiple cameras.
I tested in Unity 5.3. Perhaps the native implementation in 5.4 supports multiple cameras?

I've done it with custom shaders, but the controller models are full of edges that when you glow the outline it's not very nice looking.

Can use SkinnedMeshRenderer to change the controller colour

Trello Card https://trello.com/c/BRJRrzrH

Kinda completed with: https://github.com/thestonefox/SteamVR_Unity_Toolkit/commit/1c7c93b8349b7b1d12510792d4cce8f43d1eb988
I'm closing for now


Raft exploit fixes

Fixes for Raft Protocol incident:

Fixed formatting issue where quotes shows in titles on the website
Deleted TornadoCash link in attacker's section, it was only hash of tx and is was not relevant to the section
Fixed malicious transaction link that refers to right one and it's exact time
Included missing headers as attack types and entity types

@aholman0725 Please, review the fixes. They are relevant to your other submissions as well.

/articlecheck

@aholman0725 Please, review the fixes. They are relevant to your other submissions as well.

I will review and correct the other pull requests.

/payout -x 2


I can't use it in Django 1.11.3, it needs secure

django-cookies-samesite version:0.4.0
Django version:1.11.3
Python version:3.6.4
Operating System:centos7

Description

Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.
What I Did
I can't use this middleware because as the picture said
Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

The picture can't upload, it said I can't use samesite="None" attribute because I have not set Secure, it needs like "SameSite=None;Secure"


request for kpt live get

Similar to kubectl get, we can have a kpt live get command to get the status of resources in a package.

btw, kubectl get doesn't work with a kpt package with a grouping object.

@Liujingfang1 I am not sure I follow this exactly.  You'd like to get all of the resources in the package?

Yes, with this command, users can see the resources applied to cluster and their status. It might be similar to the kpt live status. In that case, we can close this issue.
The more interesting case is for targeting multiple cluster or filtering out certain resources that don't target the current cluster. This is a different problem, we can track it as a different issue.


escaped slash in namespaced packages

Describe the bug

when packages  are installed with https://registry.npmjs.org/
i see following lines in my package-lock.json
    "@ampproject/toolbox-runtime-version": {
      "version": "1.1.1",
      "resolved"<EMAIL_ADDRESS>      "integrity": "sha512-ibmw5p+0Sz+wingbX/Dyboe8a0+XDkMfFGSM7KFE0h2z3Op9MADup8ZPLeHT54Z7cYKmB6ob60FVHtQQDhEXNw==",
      "requires": {
        "@ampproject/toolbox-core": "^1.1.1"
      }
    },

but with verdaccio slash in package name is converted to %2f
      "resolved"<EMAIL_ADDRESS>
I don't mind actully and i had no any side effects because of this, it is just strange
To Reproduce

install any namespaced package and check package-lock.json
Expected behavior

slash should not be escaped
Configuration File (cat ~/.config/verdaccio/config.yaml)

Environment information
➜  ~ node -v
v12.16.0
➜  ~ npm -v
6.13.7
verdaccio 4.4.4 docker image

Debugging output

$ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks)
$ DEBUG=express:* verdaccio enable extreme verdaccio debug mode (verdaccio api)
$ npm -ddd prints:
$ npm config get registry prints:

Additional context

maybe similar to #1777

This bug also in Sinopia.

Hey, i decided to take a crack at this issue  #2193 , would appreciate your thoughts and comments


Values Validator => Proc's arity < 2 ?

While running specs with --warnings I found an interesting API endpoint under spec/grape/validations/validators/values_spec.rb with a successful test but maybe not the way we wanted.
The api is defined like this
params do
  requires :input_one, :input_two, type: Integer, values: { value: ->(v1, v2) { v1 + v2 > 10 } }
end
get '/proc/arity2'

And the test goes like this
context 'when arity is > 1' do
  it 'returns an error status code' do
    get '/proc/arity2', input_one: 2, input_two: 3
    expect(last_response.status).to eq 400
  end
end

It will pass but it also emits 2 warnings
Error 'wrong number of arguments (given 1, expected 2)' raised while validating attribute 'input_one'
Error 'wrong number of arguments (given 1, expected 2)' raised while validating attribute 'input_two'

If we take a closer look a the code

Proc's arity is expected to be < 2
each params a process individually (SingleAttributeIterator) so there's no relation between :input_one and :input_two even if it seems to be.

I was wondering if its really a bug. Also, I think we should raise more awareness with proper error instead of a warning.

Change the code to be v1 + v2 == 5, does it pass? If not it's a bug ;)

Change the code to be v1 + v2 == 5, does it pass? If not it's a bug ;)

It just doesn't work since arity is set to 1 and its not a MultipleAttributeIterator

Change the code to be v1 + v2 == 5, does it pass? If not it's a bug ;)

It just doesn't work since arity is set to 1 and its not a MultipleAttributeIterator

Obvious bug then :)


Article suggests that route can be created without specifying a site. This appears not to be possible

In the sharing routes across sites section of this article, it is suggested that to create a shared route, the user shouldn't specify a site on the route itself. This doesn't appear to be possible and site is a mandatory field while creating a new route and a route cannot be created by keeping this field empty. This statement needs to be clarified.

Thank you for your feedback. Can you please let us know which article you are referring to?


[STYLE CHANGE] "Know More" Button Styling Improvement

Description:
The design of the button is nice but it can be improved by making some design adjustments. By working on certain styling aspects of this button, I can make it look even better.
Reason for Change:
In order to make it look more realistic and attractive, I want to enhance styling of this button.
Current Styling:
The current styling is nice but it is looking more static. Wide border is taking all the user's attention.
Proposed Styling:
Styling changes like updating border design, background, font, margin, etc. to make it look even good.
Screenshots or Design Mockups (optional):

Priority Level (optional):

[ ] Low
[x] Medium
[ ] High

Timeline (optional):
1 day
Possible Implementation Approach (optional):
Styling changes in the css file.

Go for it


Limit parsing error resolution

Dear all,
these commits resolve the issue-148.
Regards

:white_check_mark: Build finished in 2m 51s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:ok_hand: All tests passed!
Tests run: 98, Failures: 0, Skipped: 0    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 2m 38s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:ok_hand: All tests passed!
Tests run: 100, Failures: 0, Skipped: 0    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 2m 17s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:ok_hand: All tests passed!
Tests run: 100, Failures: 0, Skipped: 0    Test Results

:information_source:  This is an automatic message

:x: Build failed in 1m 55s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:ok_hand: All tests passed!
Tests run: 100, Failures: 0, Skipped: 0    Test Results

Errors:
Filtered log (click to expand)

script returned exit code 1

:information_source:  This is an automatic message

:white_check_mark: Build finished in 1m 26s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:ok_hand: All tests passed!
Tests run: 100, Failures: 0, Skipped: 0    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 1m 32s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:ok_hand: All tests passed!
Tests run: 100, Failures: 0, Skipped: 0    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 2h 44m 26s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:exclamation: No tests found!
:information_source:  This is an automatic message

:white_check_mark: Build finished in 2m 16s
Build command:
mvn clean verify -B -e -Daudit -amd

:ok_hand: All tests passed!
Tests run: 116, Failures: 0, Skipped: 0    Test Results

:information_source:  This is an automatic message

:x: Build failed in 1h 40m 8s
Build command:
mvn clean verify -B -e -Daudit -amd

:exclamation: No tests found!
Errors:
Filtered log (click to expand)

Failed to kill container '97c1924e95ee9c21d8f24f84ea7041bf1f3476759dbd21c51a0b93ad331afb84'.

:information_source:  This is an automatic message


Unable to fetch values from red.xml variables

In my red.xml I have declared a variable ${path}.

When I try fetching the same in my test suite settings section.
Variables    ${path}/Generic_Configurations/Generic_Config.py
or
Resource    ${path}/Resource_Files/Keywords.robot
It does not show any error from the eclipse editor, even if i do Ctrl+Click, it traverse to the respective file.
But while execution of a testcase I'm getting below error:
Replacing variables from setting 'Variables' failed: Variable '${path}' not found.
Could u please tell me, why the path value is not being fetched from red.xml?

As far as I can see this is a RED issue. RED has it's own issue tracker where this should be reported. Alternatively you could try asking help on robotframework-users or on our Slack. The latter has a dedicated #red channel.

Thank you so much :)


Upgrade to Rector 0.12, PHPStan 1.0 and Symplify 10

bump deps
add missing StingsConverter

One more blocker

@samsonasik Just FYI, I tried to update everything and deployed to the website. The demo works even 2 hours after deploy :partying_face:
The main bug in previous PR was IMO this one:
https://github.com/rectorphp/getrector.org/pull/445#issuecomment-981681315
We're using json output format for Rector here, but the SymfonyStyle still outputed debug strings. That caused json_decode(...) to crash.
I fixed that here: https://github.com/rectorphp/rector-src/pull/1343, so the SymfonyStyle is always silent everywhere, if the output formatter is json.
That fixed the issue we had couple weeks ago.