motor does not reach max speed like old Gen2.x binary

continued from https://github.com/Candas1/Split_Hoverboard_SimpleFOC/issues/4#issuecomment-1604941688

In the past, I had problems with SimpleFOC due to the limited speed of the encoder readings. The Arduino interrupt handling code has a lot of overhead, made worse by the various processor layers. If the code handling the hall sensors is not fast enough, speed is limited by that. And we are using a 48MHz processor, pretty slow all considered.
I don't have time today, though. Over the weekend, I'll look at ways to speed up the interrupt handler and see if that improves anything.

I might be easier to slow down the hall sensor reading artifically and see if that has an effect on the motor max speed :-)

I am still not sure if the sensor alignment is good.
One way to check this is described here
Do you see/hear differences in how the motor spins in each direction with both binaries?
A wrong offset could lead to phase advance in one direction, and phase delay in the other direction.

you can hear in the video for yourself but no the sound is the same.
The current however is about 540 mA spinning "left" and about 580mA spinning "right".
So there indeed is some asymmetry
Good night ! i am tired now and i still want to finish the rewatch of https://en.wikipedia.org/wiki/The_Hunt_(2012_film)
Good dvd...

@RoboDurden can you please do a quick test
Add the following to the plaformio.ini file
build_unflags = -Os
build_flags = -Ofast

It compiles the binary with the highest level of optimizations possible, and if my theory is correct (slow interrupt), it should speed up the motor by a small but measurable amount.
Meanwhile I'm looking at the interrupt code

@robcazzaro , I would rather do that tomorrow when I have arrived at my solar warehouse. I have already packed the test setup in a box. And my notebook is occupied with 3d printing..

If the effect is small i will need UART rpm logging and I have yet to setup that again.

@robcazzaro , I would rather do that tomorrow when I have arrived at my solar warehouse. I have already packed the test setup in a box. And my notebook is occupied with 3d printing..

Of course, I didn't mean "right now" 😉 whenever you have time, no hurry
I saw a few Android apps that can measure RPMs if you stick a white tape to the wheel, maybe that is easier than enabling UART. I mean, UART is good to have, but an RPM meter app might be a quick way to see if there are differences

@RoboDurden I understand the interrupt handling code pretty well now, and there are a few hacks that would speed things up. But it requires a few changes to a few files and possibly breaking compatibility with the official GD32 core.
If, whenever you have time, you can confirm that compiling with -Ofast (the platformio.ini changes above) makes a difference, I will send you the modified files. Otherwise there's no point in having you waste time with the changes.

Okay that were my changes:
;build_flags = -D __PIO_DONT_SET_CLOCK_SOURCE__
;	-D __SYSTEM_CLOCK_48M_PLL_IRC8M_DIV2=48000000
build_unflags = -Os
build_flags = -Ofast

I did not hear a higher speed and android apps do not use camera but inernal IMU and you have to put your phone on the turntable..
But current was sligtly less. before 590mA and 550mA (left/right) and with your requested changes: 550mA and 500mA
So i recorded the audio: www.robosoft.de/forums/rpmsound.mp3
first you hear simpleFOC-old left/right, than your new ini file left/right, and last the gen2.x left/right
And i made a fft analysis from the constant part of all three left(?) sounds:

So yes, your requested ini changes result in a 476 Hz / 456 Hz = 4.3% speed increase,
but the gen2.x still has another 547 / 476 = 15% rpm increase on top of your changes.
This was a long sunday for me. Got 3.5 kW old solar panles for free:

Good night and good luck :-)

That is really good news @RoboDurden. We can confirm it's a IRQ overhead issue. I will send you a couple of file to try before my end of day (overwriting the existing one for now). Thanks for doing this, and clever use of the audio, well done!
Just FYI: the GD32 Arduino core has some weird behavior with the interrupts, I opened an issue in their repository to be sure https://github.com/CommunityGD32Cores/ArduinoCore-GD32/issues/105.
TLDR:
The GD32 interrupt handler (at a hardware level) groups GPIO 0-1, GPIO 2-3 and GPIO4-15 as separate interrupt source. So the EXTI01 handler must check if GPIO0 or 1 fired. Same for EXTI23. The problem is that any interrupt for pins 4 to 15 fires only one interrupt, and the handler must scan all 12 possible sources to see which one fired. So the HALL sensor on PB11 takes a long time, and the one on PC14 even more. The easy hack is to only check for pin=11 or pin=14 in the EXTI4-15 handler (which makes the code much, much faster), but I wanted to fully understand the GD32 logic for multiple ports and pins before messing with it.
Also opened https://github.com/CommunityGD32Cores/ArduinoCore-GD32/issues/106 to see if we can speed up the IRQ handler without having to change the GD32 Arduino files
Bottom line: as long as we know what interrupts we need for GPIO pins change, we can easily speed things up, and I have working code. Just want to finish testing it

Ok, here are 3 files to replace whenever you have time @RoboDurden
C:\Users[yourname].platformio\packages\framework-arduinogd32\cores\arduino\gd32\gpio_interrupt.c
[project_dir].pio\libdeps\GD32F130C8\Simple FOC\src\sensors\HallSensor.h
[project_dir]..pio\libdeps\GD32F130C8\Simple FOC\src\sensors\HallSensor.cpp
Please note that one is in the platformio GD32 platform directory, the other 2 in the project directory. And please check that the changes are compiled. Incidentally, the VS Code UI sometimes gets confused by all the nested defines, so in the gpio_interrupt.c file, the portion following `#elif defined(GD32F3x0) || defined(GD32F1x0)" is shown as disabled (greyed out), but it's active and gets compiled (GD32F1x0 is defined). It's a VS code UI bug. That one is the file I hope not to have to change thanks to this
Please also keep the -Ofast optimization enabled for now. It will make debugging harder, but a 4% gain is non-trivial. There are further optimizations possible in the Hall sensor code (updateState()), but those require more thinking. The ones I did for now are pretty safe.
If anything is unclear, just ask. I tested everything and it seems to work as expected, but it would not be the first time I screw up 😊
hall_IRQ.zip

BTW: I have been looking at other ways to potentially speed things up, but would require changes here and there. If the interrupt enhancements are not enough, I have a few more things to try. But the GD32 at 48MHz is slower than most other processors recommended for SimpleFOC, so we might have to hand-optimize more. To be fair, the FOC algorithms are more compute-heavy than the type of control implemented by most hoverboards

Don't be pessimistic, I am sure we will get there.
We are not doing foc yet, but I assume that the boards with current sensing had foc implemented.
We for sure will have to optimize because of the arduino-gd32 and simplefoc overhead.

no, the new three file do not make a difference:

The gen2.x also has a speed difference with left/right, but only 0.7%.
the simpleFOC has 1.5%.
I think the gen2.x has no interrupts for the three hall sensors at all but simply reads the state in the bldc.c 16 kHz CalculateBLDC():
	// Read hall sensors
	hall_a = gpio_input_bit_get(HALL_A_PORT, HALL_A_PIN);
	hall_b = gpio_input_bit_get(HALL_B_PORT, HALL_B_PIN);
	hall_c = gpio_input_bit_get(HALL_C_PORT, HALL_C_PIN);

So i do not really think that the three hardware interrupts from HallSensor class take longer or are less accurate ?
The EFeru FOC is faster than the Gen2.x anyway and with field weakening it can get even faster (20%). So maybe we should procceed with current sensing to get simpleFOC foc working and then have new options to tune speed and power consumption.

Foc is actually slower than trapezoidal or sinepwm, even with EFeru's firmware.

Bummer. And thanks @RoboDurden for your additional tests.
Clearly, even if the interrupt code was inefficient, it did not consume enough cycles to be a meaningful difference. Was a low hanging fruit, but not the right one.
There might be other opportunities (for example the sensor.updateState() function is over complex and calls bloated functions (like micros() that is needlessly complex, would be easier to re-implement using DWT_CYCCNT and knowing that the core clock is 48MHz). @RoboDurden if you try following the hall sensor code, you will see that a simple 3-line GPIO read in the gen2.x software is more than 50 lines, called each hall interrupt (open and close). SimpleFOC abstraction makes things more complicated (SimpleFOC can for example use SPI encoders for angle control, and the same code works with hall sensors)
As for the speed difference, is 0.7%. We are running the GD32 suing the internal oscillator, which has a 1% accuracy, so that could explain the small difference you are seeing.
The next step unfortunately it's profiling the code. The best way is to use SWD/SWO, but if you have a cheap STLink Chinese clone, the SWO pin is not enabled. It can be enabled with https://lujji.github.io/blog/stlink-clone-trace/ and it's well worth it, but time consuming and you need to be good at soldering on the small STM32 pins. With that enabled, you can ten use the STM32CubeIDE from ST to run the SWV Statistical Profiling (it's under Window, Show View, Other

That would get you a complete profiling profile, but might be too slow over SWO and pretty hard to get running.
Otherwise use macros and DWT_CYCCNT to measure execution cycles by function, which is probably the best way to go for us here. There are even available implementations like https://github.com/Serj-Bashlayev/STM32_Profiler. DWT_CYCCNT is a standard M3 core register, so available on the GD32. That library prints using SWO (once again), but it could also print to a debug output after a fixed time profiling.
Or it would be easy to enable DWT_CYCCNT  once at the beginning, create a structure for the various functions we use, store the DWT value upon entering the function, subtract the value at the end from the start, and add the instruction count to the right element of the structure for that function
//the following code needs to run only once when enabling the profiling

// enable DWT
CoreDebug->DEMCR |= 0x01000000;
// Reset cycle counter
DWT->CYCCNT = 0;
// enable cycle counter
DWT->CTRL |= 0x1;

// for each function
uint32_t start = DWT->CYCCNT;

// code to be measured here

structure.this_element += DWT->CYCCNT-start;

Then stop the code after, say, 1000 loops and print the values in the structure
Alas, I can't profile on the Bluepill dev board. Without a motor and hall sensors, the code won't work as expected. If you think you can run the profiler, I'd be happy to then analyze the code and see where we can gain. If you can't, I'll have to set up my only board and hope for the best

Well all the many lines you wrote here about such a profiler does not make me wanting to do it at all :-/
My dso single channel did show that the simpleFOC and gen2.x waveforms are quite alike.
So.why not find a clever way to see what is different...
I think candas mentioned a different alignment.
I think I have a cheap multi channel logic analyser in my train station but I forgot to take it with me to my solar warehouse.
Logging the three hall inputs and some gate outputs might shed some light in what simpleFOC is doing differently or badly.
If there are some delay parameters we could program an Auto-Tuning that optimizes rpm with a steepest descent over all parameters..
I would prefer some direct approach instead of blindly increase analyzing..
Good night for today's

Profiling is not blindly increasing analyzing 😉. It's looking for inefficient areas of the code that are holding back performance. If we are performance limited now, it's only going to get worse once we add the more complex elements of the SimpleFOC code. Insufficient processor performance is one of the most common causes of problems reported in the SimpleFOC forums.
You can't examine external signals to see where the problem is. Let's assume we find that the PWM waveform is 3% delayed compared to other code. How would we go about changing that, if we don't know what's causing it?
The main take away from my previous comment is that measuring performance is as easy as adding 2 instructions, one at the beginning, one before the exit of each function, plus half a dozen at the end of setup(). If we use macros for the beginning/end of each function, we can enable/disable perf measurement during development. And that will benefit everything we do and help optimize in the future.
That is the clever way to see what's happening, IMHO. But I'd be happy to be proven wrong, if you can find something else that works. And, yes, it might be just a hall sensor alignment, that would be great. I'll hold on doing anything wrt profiling until you fully evaluate that option.

Just a reminder.
Gen 2.x is running at 72Mhz, the simpleFOC one at 48Mhz.

okay, max speed is even slightly higher (549 Hz left and 549 Hz right) then Gen2.x when setting these to init values to 20 instead of 26 (Volt):
  driver.voltage_power_supply = 20;
  driver.voltage_limit = 20;

But the current high rises to 850-900 mA instead of the Gen2.x with less than 500 mA  :-/
I have the feeling that the problem is not code optimization but wrong pwm alignment and therfore inefficient motor control.
But with my 1 channel dso i fear i can not dig deeper.

short note to @Candas1 , that add_RTT_task.py needs
   "\"$PROJECT_PACKAGES_DIR/tool-openocd/bin/openocd\"" +
to handle path names with " " spaces.
Nice to have log output without another cable. But the sensor.getVelocity() output is not smooth at all:
GD32: 26500     0: 0    1: 1    2: 1    angle: 103.81   speed: 41.65
GD32: 26600     0: 1    1: 0    2: 0    angle: 107.79   speed: 38.98
GD32: 26700     0: 0    1: 1    2: 0    angle: 111.70   speed: 41.68
GD32: 26800     0: 0    1: 0    2: 1    angle: 115.61   speed: 35.01
GD32: 26900     0: 1    1: 1    2: 0    angle: 119.59   speed: 40.85
GD32: 27000     0: 0    1: 1    2: 0    angle: 123.43   speed: 41.02
GD32: 27100     0: 1    1: 0    2: 1    angle: 127.41   speed: 36.84
GD32: 27200     0: 0    1: 1    2: 0    angle: 131.39   speed: 41.31
GD32: 27300     0: 0    1: 1    2: 1    angle: 135.23   speed: 41.83```

Do not think that the rpm at max speed changes by 40.85/35.01 = 17% within 100ms

Maybe we should indeed implement/derive our own HallSensor class :-/

@RoboDurden the sensor is not filtered in SimpleFOC and very "spiky". There is a new smoothedsensor class being developed for that reason.
The folks running the forum recommend using motor.shaftVelocity() instead of sensor.getVelocity(). That is filtered and smoothed
As for your question on the clocks, the GD32 (like STM32) have multiple buses and clocks, each with PLLs and dividers, and it's a puzzle trying to understand how fast each bus runs. for STM32, there is a function in the STM32CubeIDE that helps calculate each and every PLL and divisor and gives you an idea in MHz or kHz. I haven't checked, but if Gen2.x is running at 72MHz, all bets are off with clocks and timing accuracy and I doubt it's. But if you want to truly understand the actual values, you need to look at the clock initialization and note each and every value being configured, then you can know the actual speed of a specific bus or component (in some cases, the speed can be half of the clock, depending on the edge polarity). Just to give you a sense, here's the clock tree for the GD32F130 (page 79, User manual)

@RoboDurden one more comment re:
void loop()
{
  // main FOC algorithm function
  // the faster you run this function the better
  // Arduino UNO loop  ~1kHz
  // Bluepill loop ~10kHz
  motor.loopFOC();
  delay(1);

Adding a delay(1) is really a lot in what should be a tight loop. A loop() with only a single instruction of delay(1) runs at less than 1kHz, which is way too low for SimpleFOC. So adding a huge delay of 1msec in that loop will never work. For a more relevant test you might want to use delayMicroseconds(1); If that causes problems, that means that the processor is not fast enough with the current code. If even something like delayMicroseconds(10) causes no problems, that means that the code is fast enough and the problem is not code efficiency (but settings and hall alignment)
The goal is to run the loop at at least 10kHz to have a smooth performance. Alas, SimpleFOC is not timer-driven, but runs in a tight loop which is a bad way to write time-critical code.
From the Gen2.x code I can't understand how the GD32 clock is initialized. Usually there is a call to SystemClock_Config() to initialize the GD32 clock, but the Gen2.x code uses a non standard way.
	//SystemClock_Config();
  SystemCoreClockUpdate();

I don't have Keil, so can't quickly test. I could port the code to VSCode, but it will take some time.
@RoboDurden If you have spare time, could you print the value of SystemCoreClock ? That variable is the GD32 clock (48000000 or 72000000). Knowing it will help making comparisons to the gen2.x code

We are free to move loopfoc and move functions to interrupts if we want to
https://community.simplefoc.com/t/simplefoc-theory-timing/2310/2?u=candas1

Good point @Candas1. Just a note that if we do that, we need to change how the hall sensor code works and disable those interrupts. Basically read the hall sensor upon receiving the timer interrupt like Gen2.x does (which would also make the velocity calculation much simpler and faster compared to the current one using the time between changes to calculate velocity, which as Robodurden noticed is very spiky). And make sure that the ADC code doesn't rely on interrupts either (Gen2.x uses ADC interrupts, but there are ways to start the ADC without interrupts). And look at how USART uses the interrupts in the Arduino code
Otherwise we have to deal with re-entrant interrupts and that causes all sorts of problems.

Yes we will trigger inserted adc at timer update event.

Some good advices were shared just now
https://community.simplefoc.com/t/hall-sensor-bldc-motor-does-not-turn/3046/10?u=candas1

According to the manual the GD32F13x is a 72 Mhz chip:

@RoboDurden not sure where you found that, but the datasheet has only 48MHz for the GD32F130xx https://www.gigadevice.com.cn/Public/Uploads/uploadfile/files/20230314/GD32F130xxDatasheetRev3.7.pdf and in this one https://www.gigadevice.com.cn/Public/Uploads/uploadfile/files/20230209/GD32F1x0_User_Manual_EN_Rev3.6.pdf it says Note:(1) The maximum operating frequency of GD32F150xx is 72 MHz, and the maximum operating frequency of GD32F130xx is 48 MHz.
It looks like 72MHz was corrected in a revision 3.2 1. Modify 72MHz system frequency to 42MHz. Jul.25, 2019. The GD32F150 runs at 72MHz, so maybe that's where the confusion comes from. It's entirely possible that most of the times it will work at 72MHz, but given how critical the correct working of the firmware is, I would not like to take that risk

GD32F130K6U6_C81552.pdf

Well, as you can see, that is an old revision from 2018. After 2019 all versions have the 48MHz clock

@Candas1 i will delete my original Split_Hoverboard_SimpleFOC repo so i can fork your repo, add the LowsideCurrentSense code and make a pull request. Then @robcazzaro if he wish can start with a SimpleFOC/src/current_sense/hardware_specific/gd32/gd32_mcu.cpp. If not, i would try to port the Gen2 analog read of
typedef struct
{
  uint16_t v_batt;
  uint16_t current_dc;
} adc_buf_t;

by adding the two low-side currents like
typedef struct
{
  uint16_t v_batt;
  uint16_t current_dc;
  uint16_t current_B;
  uint16_t current_C;
} adc_buf_t;

As the Gen2 code does read v_batt and current_dc at 16 kHz (or 32 kHz?), i think that should also be good for the LowsideCurrentSense class.

I think we need to do something different.
We would sample currents with inserted adc, and read anything else separately. We don't need battery voltage 16000 times a second, it will only delay the current sampling and foc calculation.

google gives me no hits for "inserted adc" or "inserted adc stm32" or "inserted adc gd32" :-/
That Gen2-FOC-port you host also adds the low-side currents to the one adc_buffer:

typedef struct
{
  uint16_t v_batt;
  uint16_t current_dc;
  uint16_t phase_a;
  uint16_t phase_b;
  uint16_t mcu_temp;
} adc_buf_t;

https://github.com/Candas1/Hoverboard-Firmware-Hack-Gen2/blob/master/HoverBoardGigaDevice/Inc/defines.h#L49C1-L57C1
Here the GD32 init code: https://github.com/Candas1/Hoverboard-Firmware-Hack-Gen2/blob/master/HoverBoardGigaDevice/Src/setup.c#L368
With simpleFOC i think, the current_sense is synced to the motor which is simply triggered in the loop() :-(
What might work is to not  motor.linkCurrentSense(&current_sense); at all and instead do the motor.loopFOC(); from our LowsideCurrentSense implementation when the ADC scan sequence is finished and DMA_Channel0_IRQHandler(void) gets called :-)
That would be my approach without knowing what "inserted adc" is.

@RoboDurden Gigadevice calls it inserted ADC, STM calls it injected ADC. It's a way to have multiple channels start ADC when an event (e.g. a timer) triggers it, without software assistance. SImpleFOC uses it to ensure that the current sensing is always time-aligned with the PWM. Adding interrupt servicing to the code will make it non deterministic (compared to a purely HW-based solution like injected ADC).
Here's a relevant snippet from the SimpleFOC forum

So at the moment, to avoid potential problematic situations due to the ADC conversion lasting to long and entering the high-side activation of certain mosfets we suggest using the PWM frequencies that do not exceed 20kHz. This in most cases results in behavior.

I'll leave it to you two to decide the best way to structure the high level code, you both have a lot more experience with motor control. I'll be ready to help on anything once the main structure is in place.

Injected adc is also what the simpleFOC current sense driver is using.
I have experience with injected adc and know what to do, so please leave it to me, I will check when I am back.
I think there is a lot more in the to-do list.
And let's discuss adc in a new thread.
Are we done with this thread?
Now we are able to go as fast as gen2.X,  but not at maximum target value?

max speed like Gen2 is reached but at the cost of 850-900 mA instead of the Gen2.x with less than 500 mA :-/
If the Gen2 code has a motor efficiency of 80%, then with simpleFOC we have 42% :-(
I wonder how max speed (with no load) can be reached with such a low motor efficiency.
But when moving from the NiklasFauth (that Gen2 is based on) to EFeruFOC it was like having new motors in my solar vehicle: less sound, more torque, less current (indeed i never tested for max speed).
So maybe Gen2 is below 80% efficieny.
So no, we are not really able to go as fast as gen2.
P.S. i deleted my original Split_Hoverboard_SimpleFOC, did fork your version here, cloned my fork to Github-Desktop, made some code improvements and added current_sensor code and now Github-Desktop hangs and i can not push my changes to my fork (in order to make a pull request).
I guess deleting my original repo to replace it with your fork of my original now causes an infinite loop :-(
Ideas welcome :-)

I am still betting on sensor alignment.
Current sensing/foc won't solve this issue.
Current sensing/foc won't be beneficial without the precise angle the next release will bring with sensor smoothing.

I wrongly thought that current sensing somehow replaces the hall sensor angle and gives more precise angle ?
But the lowside current sensing will be worth nothing without a precise angle derived from a hall sensor smoothing ?
With the current HallSensor and it's three channel callbacks, the hall position is immediately updated i guess - but the pwm calculating motor.loopFOC(); is not called peridically but from every 112 ms up to 1122 ms :-(
I fear that  sensor alignment can not really work that way and we better sync Hall + Current + Pwm to one 16 kHz timer.
But you say that with hall sensor smoothing, the motor.loopFOC(); will get more precise angle and it only matters that the loopFOC is called often enough.
Then hall sensor alignment (to phase pwm) is not a problem because only one precise angle is important ?
Good to hear that the simpleFOC community is already working on such a hall sensor smoothing.
But current-sensor to phase pwm alignment will be very important, so we could continue with issue #6

Without smoothing, simplefoc only has one angle per hall position.
With smoothing, the angle will be extrapolated between 2 hall positions.
FOC needs that angle, and phase currents.
Smoothing wasn't available before because simplefoc was mainly used with encoders, they are much more precise than hall sensors already.
I think the hall sensor support in simplefoc is not very old, so there is for sure room for improvement and contributions.
Sensor alignment initialization doesn't seem very accurate so we need some manual finetuning. This is not happening in the loop, and just gives you a zero electrical angle offset.

A motor is an electric magnet, using the 3 phases, you generate a 360 degree magnetic angle. This angles doesn't match with angle you get from hall sensor.

So you say that the angle doesn't match (you call it "not aligned") because the hall sensor is discrete and we need a smoothed hall sensor output. Then such a precise angle will really describe the current motor position and the motor.loopFOC will work better.
So you do not mean an alignment of digital rising/falling edges but simply a more precise motor position.
That has nothing to do with timers or interrupts :-)
But then still i wonder why Gen2.x is doing it so much better ???
Gen2 is not even updating the hall inputs with rising edge interrupts but simply samples them and does not smooth them either.

Your question was about the alignment, the precise angle is another topic.
Forget it, there is a lot of litterature about that, I can't summarize it all here.
The gen2.x firmware is only for hoverboards so it's already tuned.

Okay i quickly added a linear interpolation which should also have an effect (on max speed) if sensor smooth would be the problem -  yes i see a higher max speed
4° coarse angle: 	459 Hz / 471 Hz
with linear interpolation	477 Hz /  486 Hz
That is a 3.5% increase in max speed. Current consumption is also about 50m less and quite like the Gen2.
But Gen2 reaches 547 Hz at the same current consumption.
199: 538        angle: -16.48   speed: 0.00     51325: -0.07
215: 501        angle: -15.50   speed: 5.36     13013: 0.07
200: 1215       angle: -10.40   speed: 0.00     4477: 0.07
216: 497        angle: -1.12    speed: 23.55    2965: 0.07
215: 1215       angle: 12.01    speed: 31.04    2249: 0.07
213: 543        angle: 28.00    speed: 30.01    2326: 0.07
227: 526        angle: 44.12    speed: 33.99    2054: 0.07
227: 547        angle: 60.18    speed: 31.40    2223: 0.07
228: 545        angle: 76.24    speed: 32.53    2146: 0.07
228: 1228       angle: 92.29    speed: 34.16    2044: 0.07
227: 549        angle: 108.49   speed: 35.21    1983: 0.07
229: 1226       angle: 124.41   speed: 30.82    2265: 0.07
228: 562        angle: 140.60   speed: 35.05    1992: 0.07
227: 544        angle: 156.66   speed: 32.44    2152: 0.07
228: 1228       angle: 172.86   speed: 31.25    2234: 0.07
226: 550        angle: 188.91   speed: 33.06    2112: 0.07
216: 540        angle: 204.90   speed: 29.80    2343: 0.07
215: 532        angle: 218.59   speed: 0.00     2732: 0.07
201: 1217       angle: 228.57   speed: 15.07    4634: 0.07
216: 532        angle: 234.36   speed: 7.86     8882: 0.07
201: 538        angle: 235.83   speed: 0.00     41090: 0.07
215: 486        angle: 234.92   speed: -6.10    11438: -0.07
217: 538        angle: 229.76   speed: -15.66   4459: -0.07
213: 1228       angle: 220.40   speed: -21.47   3252: -0.07
217: 1228       angle: 207.21   speed: -28.45   2454: -0.07

The first numbers of the last colum are the micro seconds between each change of the hall sensors. Even at max speed, there is a 2 ms gap that gets filled with my linear interpolation. And as the motor speed changes slowly, the linear inerpolation should already give a far more precise angle then the coarse 0.07 deg steps.
The 0.07 degree is 2 * pi / (4°/360°). and as our hoverboard motors generate 90 ticks per revolution, that is 4° per hallsensor callback :-)
Here my simple linear interpolation:

float HallSensor::getMechanicalAngle()
{
  long iTimeSinceLastUpdate = _micros() - aiTimeLastUpdate[0];
  float fGradient1 = (float)(aiAngleLastUpdate[0]-aiAngleLastUpdate[1]) / (aiTimeLastUpdate[0]-aiTimeLastUpdate[1]);
  long fAngleSteps = aiAngleLastUpdate[0] + (long)(fGradient1 * iTimeSinceLastUpdate);
  return ((float)(fAngleSteps % cpr) / (float)cpr) * _2PI ;

  return ((float)((electric_rotations * 6 + electric_sector) % cpr) / (float)cpr) * _2PI ;
}

Maybe i could now "fine tune" by adding a 0.00x angle offset so the motor.loopFOC() -> setPhaseVoltage() -> driver->setPwm(Ua, Ub, Uc) sees a slightly advanced motor angle.. But i think their FOC algorithms should be exact and not need such a tweaking
I am not that eager to advance from my linear interpolation to a more general https://de.wikipedia.org/wiki/Spline-Interpolation

Yes, asking Bing.chatGPT "Schreibe mir in der Programmiersprache C den vollständigen Code für eine Polynominterpolation nach dem Horner-Schema welches auch die zugehörigen Koeffizienten berechnet."

 Punkt ein (x y): ", i+1);
    scanf("%lf %lf", &points[i][0], &points[i][1]); // Lese die x- und y-Werte des i-ten Punktes ein
  }

  printf("Geben Sie den Wert von x ein: ");
  scanf("%lf", &x); // Lese x ein

  double *a = coeff(n, points); // Berechne die Koeffizienten des interpolierenden Polynoms

  y = horner(n, a, x); // Berechne den Wert des interpolierenden Polynoms an der Stelle x

  printf("Der Wert des interpolierenden Polynoms an der Stelle %lf ist %lf\n", x, y); // Gib das Ergebnis aus

  free(a); // Gebe den Speicher für a frei
  for (int i = 0; i <= n; i++) {
    free(points[i]); // Gebe den Speicher für jede Zeile von points frei
  }
  free(points); // Gebe den Speicher für points frei

  return 0;
}

Sorry, i was not able to write the corresponding english prompt. This did not work:
"Write complete code for a polynomial evaluation according to the Horner's method in the C programming language, which also calculates the associated coefficients."

@RoboDurden please also keep in mind that SimpleFOC has already implemented a smoothed hall sensor class, discussion here https://community.simplefoc.com/t/smoothingsensor-experimental-sensor-angle-extrapoltion/3105/4 (I think @Candas1 also mentioned it before)
The code is in their tree, in the dev branch, this is the relevant check-in https://github.com/simplefoc/Arduino-FOC/commit/510527ae56fb70c6cf534e0a156e3c3c1f149046. It's not been released yet, but we could start using it and give feedback

!! Eureka !! - i have found it - after replacing the integer mod % with fmod, my simple linear interpolation works - exceptionaly good:
old HallSensor.cpp		460 Hz / 454 Hz at 650 mA
with 30% linear interplation:  	525 Hz / 529 Hz at only 380 mA
with 50% linear interplation:  	565 Hz / 561 Hz at only 400 mA :-)
with 60% linear interplation:  	579 Hz / 491 Hz at 480 mA :-(
Gen2				543 Hz / 544 Hz at 460 mA

Yes, the motor sometimes got stuck when changing rotation direction and makes an aweful sound :-(
But at low speed it is okay to simply disable the linear interpolation, and then that bad behavior disappeared :-)
I fear that the simpleFOC community only went for smoothing with a low pass filter. That is no prediction ..
My linear prediction might be to bold and only adding 0.5 of the linear gradient seems to be best.
Applying it at 100% will not make the motor spin correctly :-/
I still get some "ticks" at full speed that also happen on rotation changes. I guess when my prediction was to advanced and the next hall sensor event corrects this.
So a quadratic interpolation might be even better. Or indeed adding some simple 0.6 * fOld + 0.4 * fNew low pass..
The chatGPT code seems to get stuck in an infinite loop.. will still be fun to analyze that.
float HallSensor::getMechanicalAngle()
{
...
  if ((aiAngleLastUpdate[0]-aiAngleLastUpdate[1]) * (aiAngleLastUpdate[1]-aiAngleLastUpdate[2]) == 1) // no direction change
  {
    iTimeSinceLastUpdate = _micros() - aiTimeLastUpdate[0];
    fGradient1 = (float)(aiAngleLastUpdate[0]-aiAngleLastUpdate[1]) / (float)(aiTimeLastUpdate[0]-aiTimeLastUpdate[1]);
    float fAngleSteps = aiAngleLastUpdate[0] + (fGradient1 * iTimeSinceLastUpdate * 0.5);
    fAngleLin = (fmod(fAngleSteps,cpr) / (float)cpr) * _2PI ;
    return fAngleLin;
  }
  fAngleOrg = ((float)((electric_rotations * 6 + electric_sector) % cpr) / (float)cpr) * _2PI ;
  return fAngleOrg;
}

It really looks as good interpolation was the problem,
and we now have a better motor then with Gen2, Alleluja :-)

Well this FOC code is still a black box to me:
void BLDCMotor::setPhaseVoltage(float Uq, float Ud, float angle_el)
{
...
    case FOCModulationType::SinePWM :
      // Sinusoidal PWM modulation
      // Inverse Park + Clarke transformation

      // angle normalization in between 0 and 2pi
      // only necessary if using _sin and _cos - approximation functions
      angle_el = _normalizeAngle(angle_el);
      _ca = _cos(angle_el);
      _sa = _sin(angle_el);
      // Inverse park transform
      Ualpha =  _ca * Ud - _sa * Uq;  // -sin(angle) * Uq;
      Ubeta =  _sa * Ud + _ca * Uq;    //  cos(angle) * Uq;

      // center = modulation_centered ? (driver->voltage_limit)/2 : Uq;
      center = driver->voltage_limit/2;
      // Clarke transform
      Ua = Ualpha + center;
      Ub = -0.5f * Ualpha  + _SQRT3_2 * Ubeta + center;
      Uc = -0.5f * Ualpha - _SQRT3_2 * Ubeta + center;

      if (!modulation_centered) {
        float Umin = min(Ua, min(Ub, Uc));
        Ua -= Umin;
        Ub -= Umin;
        Uc -= Umin;
      }
      break;
...
  // set the voltages in driver
  driver->setPwm(Ua, Ub, Uc);
}

But i guess it is very sensitive to the motor angle, whereas Gen2 directly and non-tunable "block commutates" the three phases according to the hall sensors - not much to be done wrong.
I still have not undestood why setting driver.voltage_power_supply lower as the true battery voltage makes the motor even spin faster. And now with true voltage already exceeding Gen2, setting
 driver.voltage_power_supply = 2.6 * BAT_CELLS; // power supply voltage [V]
will boost the max speed from 42.49  to 55.28 and the current still is only 550 mA.
That is a 30% speed boost !!!
The Audacity fft frequency analisys no longer works because the sound spectrum has changed.
But as i store the angle/time tuples for my linear interpolation, i also smoothed the
float HallSensor::getVelocity()
{
  long last_pulse_diff = pulse_diff;
  if (last_pulse_diff == 0 || ((long)(_micros() - pulse_timestamp) > 2*last_pulse_diff) ) // last velocity isn't accurate if too old
    return 0;

  float vel = direction * (_2PI / (float)cpr) / (last_pulse_diff / 1000000.0f);

  bNoUpdate = true;   // thread safety
  float fPulseDiff = last_pulse_diff;
  int iSum = 1;
  for(int i=1; i<PolynomialInterpolationCount-1; i++)
  {
    int iDiff = aiTimeLastUpdate[i] - aiTimeLastUpdate[i+1];
    if (  (fPulseDiff + iDiff) > 50000) // do not average over more than 50 ms
      break;
    fPulseDiff += iDiff;
    iSum++;
  }
  if (bNoUpdate)
  {
    fPulseDiff /= iSum;
    vel = direction * (_2PI / (float)cpr) / (last_pulse_diff / 1000000.0f);
  }

  // quick fix https://github.com/simplefoc/Arduino-FOC/issues/192
  if(vel < -velocity_max || vel > velocity_max)  //if velocity is out of range then make it zero
    return 0;
  return vel;
}

I think that lowering the driver.voltage_power_supply will still allow that FOC black box to generate max sine curves.
Which afterwards get clamped to square waves in
// Set voltage to the pwm pin
void BLDCDriver6PWM::setPwm(float Ua, float Ub, float Uc) {
// limit the voltage in driver
Ua = _constrain(Ua, 0, voltage_limit);
Ub = _constrain(Ub, 0, voltage_limit);
Uc = _constrain(Uc, 0, voltage_limit);
// calculate duty cycle
// limited in [0,1]
dc_a = _constrain(Ua / voltage_power_supply, 0.0f , 1.0f );
dc_b = _constrain(Ub / voltage_power_supply, 0.0f , 1.0f );
dc_c = _constrain(Uc / voltage_power_supply, 0.0f , 1.0f );
// hardware specific writing
// hardware specific function - depending on driver and mcu
_writeDutyCycle6PWM(dc_a, dc_b, dc_c, phase_state, params);
}
But that is just my guess.
That max speed now totally outperforms the Gen2 code.

The chatGPT code was a misfire. Did only produce overflow floats.
I found a nice Lagrange Interpolation the old google way.
  long iNow = _micros();
  float fAngleNow = 0;
  for(int i=0; i<PolynomialInterpolationCount; i++)
  {
    float fCoeff = 1;
    for(int j=0; j<PolynomialInterpolationCount; j++)
      if( i!=j )
        fCoeff *=  (float)(iNow - aiTimeLastUpdate[j]) / (float)(aiTimeLastUpdate[i] - aiTimeLastUpdate[j]);
    fAngleNow += fCoeff * aiAngleLastUpdate[i];
  }
  fAngleNow = 0.1 * fAngleNow + 0.9 * (float)(electric_rotations * 6 + electric_sector);    // only take 10% of the predictioin
  fAngleLagrange = (fmod(fAngleNow,cpr) / (float)cpr) * _2PI ;
  if (bNoUpdate)
    return fAngleLagrange;

But adding mor then 10% of that 4th polynomial interpolation did block the motor :-(
(i logged the fAngleLagrange and fAngleOrg and could verify that the predicted fAngleLagrange was reasonable)
I also noticed that the execution time of that code affected the motor behavior (additional 30 mA) even without using fAngleLagrange at all. So the getMechanicalAngle()  call from BLDCMotor::loopFOC() is very time critical.
And my simple linear interpolation might already be the optimum :-)

I still have not undestood why setting driver.voltage_power_supply lower as the true battery voltage makes the motor even spin faster. And now with true voltage already exceeding Gen2, setting

The way I understand it, SimpleFOC needs to know the power supply voltage value when calculating currents (you can set the motor impedance) and to optimize voltage and current in each phase.
Then you use motor.voltage_limit and driver.voltage_limit to protect the motor especially when starting to protect the motor and board.
Let's say that you have a 10V power supply and set driver.voltage_power_supply to 10 and driver.voltage_limit to 5V. The PWM will never produce more than 50% voltage when the algorithm would output 100%. If you set driver.voltage_limit to 10, now at max phase voltage you will get 10V
It looks as if, by setting the power supply value lower than actual, you are driving the PWM more fully. Do you have motor.voltage_limit and driver.voltage_limit set, by chance? Try printing those values and see if they are the same or lower than the power supply value
Keep in mind that speed is only one aspect. For my uses, for example, I will need the motor to move very slowly (<10 RPM) with constant torque and be able to hold position with the same torque. The value of using SimpleFOC, for me, would be that once a board and motor parameters are optimized, it can be used in many different ways equally easy, without having to change the core code, just change settings in the setup()

I would check the 3 voltages with this, maybe we see what happens

I am wondering if at high duty cycle the values are clamped, so the sinusoidal is flat.

I am wondering if at high duty cycle the values are clamped, so the sinusoidal is flat.

The best way to verify that would be to use a logic analyzer connected to the processor PWM pins (it's enough to only use 3 pins, like the 3 high phase pins), and once the motor is spinning at constant speed, capture enough pulses to see the PWM signal. It's easy that way to see a clipped signal, as a long sequence of 1s

Thanks @robcazzaro , having two voltage_limit at the same time in different classes looks like very bad programming behavior to me.
FOCMotor::voltage_limit is never used in that base class FOCMotor class anyway and the dereived BLDCMotor can not work without a driver and also makes multiple uses of driver->voltage_limit.
So i guess it is a bug to use FOCMotor::voltage_limit in some other places of the same code file :-(
If indeed some other derived FOCMotor class would need its own voltage_limit without having a driver, then it should be defined in that other class and not in the base class.
OUT2T( motor.voltage_limit, driver.voltage_limit)
-> 12.00: 25.20

The 25.2 Volt is my 7s 3.6 * BAT_CELLS, the 12.0 is still the DEF_POWER_SUPPLY
Indeed, in BLDCMotor::move(float new_target), the motor::voltage_limit is used for clamping:
  switch (controller) {
    case MotionControlType::torque:
      if(torque_controller == TorqueControlType::voltage){ // if voltage torque control
        if(!_isset(phase_resistance))  voltage.q = target;
        else  voltage.q =  target*phase_resistance + voltage_bemf;
        voltage.q = _constrain(voltage.q, -voltage_limit, voltage_limit);

But BLDCMotor::setPhaseVoltage concludes with   driver->setPwm(Ua, Ub, Uc); and there the
Ua = _constrain(Ua, 0, voltage_limit);
has no effect because driver::voltage_limit is way higher than the still default 12.0 used for clamping in BLDCMotor::move().
And the final clamp to [0.0..1.0] i guess will only allow the max of 1.0 if i also reduce the driver::voltage_power_supply to 12.0 V
  dc_a = _constrain(Ua / voltage_power_supply, 0.0f , 1.0f );
I think there should not be a FOCMotor::voltage_limit at all :-(
It is never used in that base class FOCMotor, and in the derrived BLDCMotor class driver->setPwm(Ua, Ub, Uc); is hard coded in BLDCMotor::setPhaseVoltage(...). So it would be no problem at all to not use FOCMotor::voltage_limit in BLDCMotor::move() (and other places) but
        voltage.q = _constrain(voltage.q, -driver->voltage_limit, driver->voltage_limit);
instead of what is now implemented:
        voltage.q = _constrain(voltage.q, -voltage_limit, voltage_limit);
I also do not (want) to undestand why using voltages when in the end we factor and clamp it down to [0..1] anyway.
The voltage_power_supply is only used to scale the duty cacle down to 0..1. There is no analog read of the battery voltage to compare it against voltage_power_supply. in neither the base class BLDCDriver nor any of the derived BLDCDriver_x_PWM classes.
The [0.0..100.0] duty cycle is more intuitive.
But i think it is too late to change that now.
At least the FOCMotor:.voltage_limit should be removed i guess.
Do not really have the happiness to start arguing with the simpleFOC community..
@Candas1 , why don't you accept my pull request: https://github.com/Candas1/Split_Hoverboard_SimpleFOC/pull/7  ?
Then @robcazzaro could already begin with adc low level stuff.
I do not really have a use scenario that needs currents sensing. I feel current sensing is NOT needed for FOC but only for torque control..

Because I don't aim to maintain my own branch of simplefoc except the GD32 drivers.
Anything else will also be beneficial to the simpleFOC community.
You are working on topics the simpleFOC team is already about to release, without even trying it.
This will even make it harder to merge anything new they are bringing.

I don't pose a problem if I can't offer any solutions

i am totally undecided whether FOCMotor::voltage_limit or BLDCDriver_x_PWM::voltage_limit should be removed.

So no solutions to offer and therefore no happiness to pose questions to other people.

@robcazzaro i still do not really understand why you don't buy some used hoverboards locally. I do not know all the US websites to buy used items but already facebooks lists multiple items: https://www.facebook.com/marketplace/seattle/hoverboards?maxPrice=30&exact=false
And me (and i guess Candas too) would be happy to each $20 donate  to you for getting some boards.
If your really plan to do multiple projects with hoverbaords you will need your own stock of used hoverboards anyway.

i am not really making progress with my linear interpolation. Has taken all day to verify that my angle prediction works quite good, but the motor does not like the "better" values.
I also notice that the motor is spinning more poorly left then right.
Sometimes it even gets "out of phase" and is rotating with a bad stutter. Then after a few seconds it kicks back "into the right phase" and rotates "nicely" again.
So the original HallSensor is also not perfect.
With Gen2 however, the phase pwm are simply directly calculated from the 6 (8?) possible hall combination.
  // Determine current position based on hall sensors
  hall = hall_a * 1 + hall_b * 2 + hall_c * 4;
  pos = hall_to_pos[hall];

So nothing can get "out of phase" over time.
With simpleFOC however, the angle increments with every hall step, is then mapped to 360° = [0.0 .. 2pi], some offset is added
sensor->getMechanicalAngle()  - zero_electric_angle
but once that angle "gets out of phase", the three pwm phase voltages
get applied wrong "forever".
I think there should be an ongoing closed loop that corrects the zero_electric_angle.
to match the coarse hall_to_pos[hall];
And i guess there should be a zero_electric_angle_left and zero_electric_angle_right
@Candas1 , how did you obtain motor.initFOC(2.09,Direction::CCW); // Start FOC without alignment which skips the BLDCMotor::alignSensor()- which does not work at all for me. Only your 2.09 work
Was that simply try and error ? Then maybe with my "better" linear interpolation, i must find a new value.

Please check those comments:
https://github.com/Candas1/Split_Hoverboard_SimpleFOC/issues/1#issuecomment-1592731286
https://github.com/Candas1/Split_Hoverboard_SimpleFOC/issues/5#issuecomment-1605016055
https://github.com/Candas1/Split_Hoverboard_SimpleFOC/issues/5#issuecomment-1610263400

@RoboDurden SimpleFOC is a bit hard to understand and has quite a few quirks. On the other hand, it's solid code that works for a variety of projects and hardware. Trying to hack around its inner workings without understanding why certain things were done a certain way it's a recipe for frustration. Many of the issues we are facing are known by the SimpleFOC team and work is happening to address them.
If we think something could be done better, the right approach is to engage with them and suggest changes. Most of the times, their answer will explain why their approach is better and the side effects of doing things differently. Every time I suggested changes, they were incredibly open and responsive.
You say I do not really have a use scenario that needs currents sensing. I feel current sensing is NOT needed for FOC but only for torque control.. Well, torque control at very low speeds is the only scenario that matters to me 😁, and the reason why we even started having these conversations. I'd like to have a solution for torque control using the hoverboard split boards. I already have other boards (like DRV8302 that works perfectly well with SimpleFOC, just wanted to see if it's possible to use a split hoverboard board
So, while I appreciate your desire to quickly get the code to a working state for higher speeds, that is not an approach I think will work long term. As I mentioned before, there is already a solution for a smoothed hall sensor that works at any speed. It might be suboptimal in certain cases compared to your approach, but works in general. IF you think your approach is better, I'm sure that the SimpleFOC people will be interested in hearing about it and discuss it with you.

Thanks @Candas1 for the links, i had searched for them but did not find them in our too long issues.
Tomorrow i will try adding the yet missing parameters to
BLDCMotor(int pp,  float R = NOT_SET, float KV = NOT_SET, float L = NOT_SET);
That would give a "better" result in BLDCMotor:move()
        if(!_isset(phase_resistance))  voltage.q = target;
        else  voltage.q =  target*phase_resistance + voltage_bemf;

Then maybe the BLDCMotor::alignSensor() will result in a zero_electric_angle that at least will also work like your 2.09
I guess i can measure R with my multimeter, simple the resistance from blue phase to yellow.
KV should be 14, according to one of my first youtube videos: https://youtu.be/pbip8CVaAbc
How can i obtain L ? My cheap multimeters do not seem to measure inductivity.

'''If the top speed is higher in one direction than the other, use sine modulation and adjust motor.zero_electric_angle until both directions are equal (it’s set automatically by calibration, but sometimes needs manual adjustment to correct for imperfect sensor placement)

Spent nearly most of Sunday to find better motor settings but haven't found anything better then the 2.09 that Candas chose.
My linear predictions still gives a 30% speed boost but the motor gets very noisy. Do not understand this.

But even so the prediction should be accurate, the motor gets very noisy.
In these tests, i let the motor spin left for about 2 seconds, then right for another 2 seconds. Total time 5 seconds and samples_LowPass times LoopFOC was called.
  if (-fSpeed > driver.voltage_limit)  // will be clamped and therefore be constant
    fSpeedAvgMax = 0.99 * fSpeedAvgMax + 0.01 * fVelocity;

Today i have built myself a Gen1 test setup. Will try to flash our simpleFOC software to it. Then i can compare with the noise and max speed of the EFeru FOC.

Would be nice to have at least overall current adc. Then i could compare the power with speed to judge the efficiency of the different firmwares. If i remember correctly, the EFeru firmware outputs speed in rpm. SimpleFOC velocity seems to be angulare frequency ( 2 pu / f ).
First test run with EFeru FOC was so silent that my Audacity frequency analysis would give no results.
As Candas will not accept my pull request, @robcazzaro if you want to and know how to start a SimpleFOC/src/current_sense/hardware_specific/gd32/gd32_mcu.cpp , you could download my preperation from my fork: https://github.com/RoboDurden/Split_Hoverboard_SimpleFOC
But i guess, Candas will be back from holidays in about a week and if he wants to do it, we can happily wait :-)

had really bad sleep last night and am very tired today.
Have problems to match the EFeru rpm
rpm_l:-368 rpm_r:0 cmdL:1000 cmdR:1000 BatADC:1016 BatV:2695 TempADC:1722 Temp:227
rpm_l:-368 rpm_r:0 cmdL:1000 cmdR:1000 BatADC:1006 BatV:2695 TempADC:1722 Temp:227
rpm_l:-347 rpm_r:0 cmdL:1000 cmdR:1000 BatADC:1004 BatV:2695 TempADC:1722 Temp:227
rpm_l:-353 rpm_r:0 cmdL:1000 cmdR:1000 BatADC:1030 BatV:2695 TempADC:1722 Temp:227
rpm_l:-359 rpm_r:0 cmdL:1000 cmdR:1000 BatADC:1016 BatV:2695 TempADC:1722 Temp:227
rpm_l:-365 rpm_r:0 cmdL:1000 cmdR:1000 BatADC:1008 BatV:2695 TempADC:1722 Temp:227
rpm_l:-359 rpm_r:0 cmdL:978 cmdR:978 BatADC:1026 BatV:2695 TempADC:1722 Temp:227
rpm_l:-326 rpm_r:0 cmdL:900 cmdR:900 BatADC:1006 BatV:2695 TempADC:1722 Temp:227
rpm_l:-305 rpm_r:0 cmdL:828 cmdR:828 BatADC:1004 BatV:2695 TempADC:1722 Temp:227```

with the simpleFOC angular speed.

`float vel = direction * (_2PI / (float)cpr) / (last_pulse_diff / 1000000.0f);  // cpr = 90 (hall ticks per revolution)`

Speed_left/right  Average_r/l  AverageDelta_l/r   linerar [%]   loopFOC[ns]_loop[ms]   samples_LowPass
-20.40: 19.83   17.20: -17.13   0.57: 0.06      fLinAdd: 92.50  213: 846        18866

The EFeru max rpm was 368. Divided by my 26 Volt confirms the 14.15 KV of my old youtube video.
So `368 / 60 = 6.13 `is the frequency f (revs). And therefore, the angular velocity `w = 2 pi / T = 2 pi f `should be 2 pi 6.13 = 38.5 which clearly is not close to 20.4 max_simpleFOC or 17.2 average_simpleFOC.

But maybe a factor 2 ?

But i do not find that in the simpleFOC code, unless we should use pole_pairs = 30 instead of 15:

Maybe there is an issue with time measurement.
I can measure the speed with a tachometer when I am back.

If it's higher than rated KV you're probably doing field weakening.

This probably explains why the motor would slow down at higher target values.

Well this does not explain why my simple linear prediction makes a 37% speed boost. Here with only
    fSpeed = (0.5*driver.voltage_power_supply)  * (ABS(	(float)(((millis()-iLoopStart)/10 + 250) % 1000) - 500) - 250)/250;
    float fLinAdd = 0.9;

rpm/V loopFOC[ms]:HallStep[ms]  fAngle0:fAngle    predict[ms]:fAngleLin ...
KV: 17.29       273: 1460       3.28: 3.35      1307:3.33 1065:3.32   69:3.28
KV: 17.38       274: 1431       3.35: 3.42       hall_int  1176:3.40  933:3.39  121:3.36
KV: 16.45       273: 1768       3.42: 3.49      1573:3.49 1331:3.48 1090:3.47  264:3.43
KV: 17.06       269: 1545       3.49: 3.56      1458:3.55 1220:3.54  980:3.53   79:3.49
KV: 17.08       270: 1452       3.56: 3.63      1322:3.61 1084:3.60  183:3.57

first column = KV [rpm/V]
second column = difference of last two loopFOC calls [ms] and milliseconds between the two last hall steps (the greater the slower the speed)
third column = the previous angle (hall pos) that the linear prediction based on: the true angle that followed after these ms between the two last hall steps (each increment is 4° = 0.07 rad)
fourth column = array of up to 8 paris predict time in ms: predicted angle.
Usually the predict times change by loopFOC[ms] but sometimes my loop() takes up to 1ms.
Here when the motor slowed down:
KV: 9.85        275: 2663       1.12: 1.19      2565:1.18 2323:1.17 2081:1.17 1839:1.16 1598:1.15 1356:1.15  197:1.12
KV: 9.77        276: 2733       1.19: 1.26      2563:1.25 2320:1.24 2077:1.24 1834:1.23 1590:1.22 1349:1.22  177:1.19
KV: 9.84        275: 2708       1.26: 1.33      2497:1.31 2254:1.31 2011:1.30 1768:1.30 1525:1.29 1282:1.29  107:1.26
KV: 9.90        275: 2587       1.33: 1.40      2444:1.38 2202:1.38 1960:1.37 1719:1.37 1477:1.36 1235:1.36   65:1.33
KV: 9.76        272: 2812       1.40: 1.47      2719:1.46 2480:1.45 2241:1.45 2003:1.44 1772:1.44 1534:1.43 1301:1.43

My linear extraplation/prediction works nicely and i don't understand why that should have a field weakening effect.
BLDCMotor:move(target) does nothing but if(!_isset(phase_resistance))  voltage.q = target;
loopFOC() does nothing with our setting TorqueControlType::voltage and directly calls setPhaseVoltage(voltage.q, voltage.d, electrical_angle);
in which i do not see any parameter that might tune the pwm output in some bad way that my linear predicition resolves:
    case FOCModulationType::SinePWM :
      // Sinusoidal PWM modulation
      // Inverse Park + Clarke transformation

      // angle normalization in between 0 and 2pi
      // only necessary if using _sin and _cos - approximation functions
      angle_el = _normalizeAngle(angle_el);
      _ca = _cos(angle_el);
      _sa = _sin(angle_el);
      // Inverse park transform
      Ualpha =  _ca * Ud - _sa * Uq;  // -sin(angle) * Uq;
      Ubeta =  _sa * Ud + _ca * Uq;    //  cos(angle) * Uq;

      // center = modulation_centered ? (driver->voltage_limit)/2 : Uq;
      center = driver->voltage_limit/2;
      // Clarke transform
      Ua = Ualpha + center;
      Ub = -0.5f * Ualpha  + _SQRT3_2 * Ubeta + center;
      Uc = -0.5f * Ualpha - _SQRT3_2 * Ubeta + center;

      if (!modulation_centered) {
        float Umin = min(Ua, min(Ub, Uc));
        Ua -= Umin;
        Ub -= Umin;
        Uc -= Umin;
      }

The electrical_angle add your 2.09 to my mechanical_angle (prediction) but that only is the neccessaary calibration of the hall sensors position.
My linerar prediction works and i can not see how that should be bad.
Here max speed without my linerar prediction:
    fSpeed = (0.5*driver.voltage_power_supply)  * (ABS(	(float)(((millis()-iLoopStart)/10 + 250) % 1000) - 500) - 250)/250;
    float fLinAdd = 0.0;

KV: 12.54       263: 2122       1.61: 1.68      1957:1.61 1724:1.61 1482:1.61 1241:1.61  240:1.61
KV: 12.62       265: 2102       1.68: 1.75      2018:1.68 1784:1.68 1549:1.68 1315:1.68 1081:1.68   99:1.68
KV: 12.71       264: 2022       1.75: 1.82      1955:1.75 1722:1.75 1488:1.75 1253:1.75  182:1.75

Of course the "predicted angle" stays at the old angle for all loopFOC calls until the next hall step occurs :-(
17.38 / 12.71 = 37% speed boost
Yes i know that we should not get a KV of more than 15 for our bldc motors.
And with full voltage_power_supply (1.5* gets clamped to 1.0) i even get a KV of 21.63 rpm/V
    fSpeed = (1.5*driver.voltage_power_supply)  * (ABS(	(float)(((millis()-iLoopStart)/10 + 250) % 1000) - 500) - 250)/250;
    float fLinAdd = 0.9;
KV: 21.49       263: 1151       1.88: 1.95       977:1.93  174:1.89
KV: 20.68       266: 1415       1.95: 2.02      1278:2.02 1042:2.01  807:2.00   90:1.96
KV: 21.46       267: 1175       2.02: 2.09      1016:2.07  130:2.03
KV: 21.63       268: 1152       2.09: 2.16      1052:2.15  815:2.14  109:2.10
KV: 20.57       266: 1402       2.16: 2.23      1197:2.23  962:2.21  169:2.17

Only by correctly predicting the current hall position :-/
I should test the motor under load.. Don't really know how to do this.
Ideas welcome.
P.S. i do not like voltage_power_supply nonsene. There isn't a single Driver class in simpleFOC that drives a motor with an analog voltage. It is always (of course) a pwm ration from 0.0 to 1.0.
So why the complexity of voltage_power_supply , voltage_limit and move(voltage) ??
Our 10s batteries can go from 42.0 Volt down to 25.0V ! But of course we always want the 0.5 for pure sine FOC.
Using absolute voltages does not make any sense to me.

So stm32 test setup is running and i can compare Gen1 vs Gen2 with the same simpleFOC software.
But i have a different motor in that setup :-/ It is not impossible that the Gen2 test setup has 20mm magnets and the Gen1 test setup has 25mm magnets inside. For difference see my old test: https://www.youtube.com/watch?v=cYIpnW6RCSE
And we really need a 16 kHz timer to move loopFOC() out of the main loop(). I can not log more output via UART on the Gen1 without the motor spinning poorly because loopFOC does no longer get called often enough.
Gen1 ----------------------
  driver.voltage_power_supply = 3.6 * BAT_CELLS; // power supply voltage [V]
  driver.voltage_limit = 1 * driver.voltage_power_supply;   // 0.3 = keep well below 1.0 for testing !
    fSpeed = (0.5*driver.voltage_power_supply)  * (ABS(	(float)(((millis()-iLoopStart)/10 + 250) % 1000) - 500) - 250)/250;
    float fLinAdd = 0.0;

KV: rpm/V   latest loopFOC[ms]: max loopFOC[ms]
KV: -9.58  186: 2615
KV: -9.53  183: 2835
KV: -10.34  187: 2531
KV: -11.10  183: 2264
KV: -11.26  187: 2200
KV: -12.00  188: 2040
KV: -11.84  210: 2227
KV: -12.11  176: 2205
KV: -11.38  173: 2260
KV: -11.26  173: 2333
KV: -10.23  204: 2565
KV: -10.16  177: 2574
KV: -9.54  173: 2708

  driver.voltage_power_supply = 3.6 * BAT_CELLS; // power supply voltage [V]
  driver.voltage_limit = 1 * driver.voltage_power_supply;   // 0.3 = keep well below 1.0 for testing !
    fSpeed = (0.5*driver.voltage_power_supply)  * (ABS(	(float)(((millis()-iLoopStart)/10 + 250) % 1000) - 500) - 250)/250;
    float fLinAdd = 0.7;

KV: -8.48  187: 3060
KV: -8.78  188: 3014
KV: -9.47  182: 2793
KV: -10.09  188: 2435
KV: -10.19  187: 2505
KV: -11.08  183: 2329
KV: -10.96  183: 2347
KV: -11.76  188: 2289
KV: -11.59  178: 2283
KV: -11.95  188: 2258
KV: -11.23  173: 2286
...
KV: 8.23  207: 3047
KV: 8.66  205: 3058
KV: 9.94  233: 2647
KV: 10.33  200: 2583
KV: 11.26  207: 2323
KV: 12.09  235: 2305
KV: 12.51  206: 2237
KV: 13.36  207: 2057
KV: 14.25  207: 1875
KV: 14.71  207: 1871
KV: 15.00  205: 1921
KV: 16.38  206: 1526
KV: 16.37  226: 1610
KV: 16.84  207: 1592
KV: 16.83  206: 1555
KV: 15.65  205: 1673
KV: 15.24  207: 1601
KV: 13.90  207: 1989
KV: 13.48  208: 1817
KV: 12.63  207: 2149
KV: 11.54  206: 2356
KV: 11.20  208: 2264
KV: 10.09  207: 2597
KV: 9.30  207: 2818

Gen2.0 ----------------------
  driver.voltage_power_supply = 3.6 * BAT_CELLS; // power supply voltage [V]
  driver.voltage_limit = 1 * driver.voltage_power_supply;   // 0.3 = keep well below 1.0 for testing !
    fSpeed = (0.5*driver.voltage_power_supply)  * (ABS(	(float)(((millis()-iLoopStart)/10 + 250) % 1000) - 500) - 250)/250;
    float fLinAdd = 0.0;

KV: 11.73  255: 2278
KV: 12.01  258: 2408
KV: 11.64  255: 2379
KV: 11.53  290: 2472
KV: 11.30  291: 2272
KV: 10.93  258: 2584
KV: 10.07  258: 2809
KV: 10.11  257: 2604
KV: 9.09  255: 3030
KV: 9.22  257: 2886
...
KV: -8.75  233: 3041
KV: -9.21  234: 2940
KV: -9.79  234: 2882
KV: -10.59  230: 2337
KV: -11.17  233: 2440
KV: -11.24  233: 2331
KV: -12.47  233: 1996
KV: -11.92  234: 2166
KV: -13.56  224: 1820
KV: -12.09  234: 2169
KV: -12.08  234: 2390
KV: -11.34  265: 2343
KV: -11.34  231: 2454
KV: -10.39  230: 2593
KV: -10.30  234: 2623
KV: -9.51  234: 2801
KV: -9.50  235: 2639

  driver.voltage_power_supply = 3.6 * BAT_CELLS; // power supply voltage [V]
  driver.voltage_limit = 1 * driver.voltage_power_supply;   // 0.3 = keep well below 1.0 for testing !
    fSpeed = (0.5*driver.voltage_power_supply)  * (ABS(	(float)(((millis()-iLoopStart)/10 + 250) % 1000) - 500) - 250)/250;
    float fLinAdd = 0.9;

KV: 9.78  258: 2741
KV: 10.16  258: 2603
KV: 11.68  259: 2184
KV: 11.87  290: 2290
KV: 13.33  259: 2040
KV: 13.13  255: 2096
KV: 13.97  255: 1750
KV: 15.03  256: 1726
KV: 15.38  258: 1713
KV: 15.83  258: 1726
KV: 16.96  289: 1520
KV: 16.27  247: 1542
KV: 16.29  259: 1610
KV: 16.76  287: 1531
KV: 15.66  259: 1716
KV: 15.62  260: 1569
KV: 15.36  258: 1683
KV: 14.38  258: 1822
KV: 13.99  256: 1835
KV: 12.71  258: 1913
KV: 12.56  258: 2107
KV: 10.58  241: 2650
KV: 10.51  259: 2466
KV: 9.20  292: 2751
...
KV: -8.29  234: 3082
KV: -9.04  230: 2744
KV: -9.66  230: 2602
KV: -9.68  230: 2771
KV: -10.71  235: 2499
KV: -10.63  231: 2471
KV: -11.71  234: 2254
KV: -11.65  235: 2201
KV: -12.75  230: 2062
KV: -12.40  222: 2115
KV: -12.73  230: 2118
KV: -11.68  234: 2283
KV: -12.14  230: 2134
KV: -10.83  233: 2460
KV: -11.02  230: 2258
KV: -10.08  233: 2620

The Gen2 GD32 code is about as efficient as the STM32 code :-)
My linear prediction now only seems to work in one direction for both Gen1 and Gen2. Need to investigate why that changed for Gen2.
But i also see a 33% speed boost for the stm32 Gen1 code.
So that strange boost has nothing to do with the GD32_mcu.cpp implementation.
Will have to do more test.
My code is online here https://github.com/RoboDurden/Split_Hoverboard_SimpleFOC/blob/main/src/main.cpp
But the GD32/HallSensor and STM32/HallSensor changes needed are offline.
So the code currently is not ready for a pull request

I switched back to trapezoidal, sinePWM was a mistake without interpolation, I thought it would just perform like trapezoidal and it was less noisy.
In the next few days I will try the simpleFOC dev branch improvements (interpolation, better interrupt handling, faster sin/cos, new initFOC) together with the gd32 drivers and sinePWM.

Good to hear from you Candas :-)
When can i expect a 10-16 kHz Timer Interrupt to call loopFOC ?
If this will take you more then 2 weeks i might want to start with a dummy currentSensor the Gen2 style. Only to move the loopFOC out of the main loop. Then i may try to add adc sampling like done in the Gen2 code and we can later see what works better. your inserted adc or the gen2 style.

Feel free to experiment if you want to learn.

hm, no answer to my question.

Okay thank you for that answer :-) I thought it is obvious that we need an additional interrupt for loopFOC.
So i will begin to port the Gen2 adc the next days..
I am currently working on a vertical axis wind turbine with a 10" hoverboard motor:

The idea was to only use two blades as the hoverboard controller can start the turbine..
But now i guess i will add a little savonius wind turbine of 1/4 diameter. The tip-speed-ration is only about 1/4 of the darrieus..
Don't know if it is really worth adding a hoverboard controller with a kind of mppt that varies a target speed..


Add more verbose error reporting

this is an ongoing issue, but progress has been made in:
8166632e926ba4c11a1c1181525d29ea6be1343f

The goal is to replace all C# exceptions with more descriptive custom-made exceptions.

Further work here will await #47

see also: cc6c4e1a10715d6397546fcb0de9eeed27a0cef9


[Bug] Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [x]

I have been trying to run matano in a fresh personal AWS account after having it tried it an another account with extended lambda limits to see if there exists any additional configuration / request for quota increase.  I hit upon this error with matano.
Details below.
Version : matano/0.0.0 linux-x64 node-v14.18.1
Note: This is the nightly build as of today.
Snippet Error from Terminal Below :
rams3sh@monastery:~/Garage/matano$ matano init
━━━ Matano: Get started Wizard ━━━

Welcome to the Matano init wizard. This will get you started with Matano.
Follow the prompts to get started. You can always change these values later.

✔ Which AWS Region to deploy to? · us-east-1
✔ What is the AWS Account ID to deploy to? · XXXXXXXXXXXXX
✔ Do you have an existing matano directory? (y/N) · false
  I will generate a Matano directory in the current directory.
✔ What is the name of the directory to generate?(use . for current directory) · .
✔ Generated Matano directory at /home/rams3sh/Garage/matano.
✔ Successfully initialized your account.
⠦ Now deploying Matano to your AWS account...
›   Error: An error occurred: Command failed with exit code 1: /usr/local/matano-cli/cdk deploy DPMainStack --require-approval never --app /usr/local/matano-cli/matano-cdk
...

 ›   Failed resources:
 ›   MatanoDPMainStack | 7:51:57 PM | CREATE_FAILED        | AWS::Lambda::Function            | DPMainStack/LakeWriter/AlertsFunction (LakeWriterAlertsFunctionCB567D9B)
 ›   Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [50].
 ›    (Service: Lambda, Status Code: 400, Request ID: c990af9b-a3e6-4328-a3c7-4f0b01967c4f)" (RequestToken: b6eb4fad-441b-2493-86c5-5c29b6969a6f, HandlerErrorCode:
 ›   InvalidRequest)
 ›
 ›    ❌  DPMainStack (MatanoDPMainStack) failed: Error: The stack named MatanoDPMainStack failed creation, it may need to be manually deleted from the AWS console:
 ›   ROLLBACK_COMPLETE: Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its
 ›   minimum value of [50]. (Service: Lambda, Status Code: 400, Request ID: c990af9b-a3e6-4328-a3c7-4f0b01967c4f)" (RequestToken: b6eb4fad-441b-2493-86c5-5c29b6969a6f,
 ›   HandlerErrorCode: InvalidRequest)
 ›       at FullCloudFormationDeployment.monitorDeployment (/snapshot/node_modules/aws-cdk/lib/api/deploy-stack.ts:505:13)
 ›       at runMicrotasks (<anonymous>)
 ›       at processTicksAndRejections (internal/process/task_queues.js:95:5)
 ›       at deployStack2 (/snapshot/node_modules/aws-cdk/lib/cdk-toolkit.ts:265:24)
 ›       at /snapshot/node_modules/aws-cdk/lib/deploy.ts:39:11
 ›       at run (/snapshot/node_modules/p-queue/dist/index.js:163:29)
 ›
 ›    ❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named MatanoDPMainStack failed creation, it may need to be manually deleted from the AWS console:
 ›    ROLLBACK_COMPLETE: Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its
 ›   minimum value of [50]. (Service: Lambda, Status Code: 400, Request ID: c990af9b-a3e6-4328-a3c7-4f0b01967c4f)" (RequestToken: b6eb4fad-441b-2493-86c5-5c29b6969a6f,
 ›   HandlerErrorCode: InvalidRequest)
 ›       at deployStacks (/snapshot/node_modules/aws-cdk/lib/deploy.ts:61:11)
 ›       at runMicrotasks (<anonymous>)
 ›       at processTicksAndRejections (internal/process/task_queues.js:95:5)
 ›       at CdkToolkit.deploy (/snapshot/node_modules/aws-cdk/lib/cdk-toolkit.ts:339:7)
 ›       at initCommandLine (/snapshot/node_modules/aws-cdk/lib/cli.ts:374:12)
 ›
 ›   Stack Deployments Failed: Error: The stack named MatanoDPMainStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource
 ›   handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [50].
 ›   (Service: Lambda, Status Code: 400, Request ID: c990af9b-a3e6-4328-a3c7-4f0b01967c4f)" (RequestToken: b6eb4fad-441b-2493-86c5-5c29b6969a6f, HandlerErrorCode:
 ›   InvalidRequest)
 ›   Created temporary directory for configuration files: /tmp/mtnconfigv9yADs/config
 ›   arn:aws:cloudformation:us-east-1:XXXXXXXXXX:stack/MatanoDPCommonStack/cebd94d0-7e14-11ed-9855-0e5a30013c2f

Lambda Quotas :
rams3sh@monastery:~/Garage/matano$ aws lambda get-account-settings
{
    "AccountLimit": {
        "TotalCodeSize":<PHONE_NUMBER>0,
        "CodeSizeUnzipped": 262144000,
        "CodeSizeZipped": 52428800,
        "ConcurrentExecutions": 50,
        "UnreservedConcurrentExecutions": 50
    },
    "AccountUsage": {
        "TotalCodeSize": 1337,
        "FunctionCount": 1
    }
}

Please let me know how to proceed from here.
Also, should I have to mandatory increase the lambda quota since it has a separate pricing ? Can there be any option not to have this concurrency enabled as part of matano deployment ? This will be helpful for experimentation use cases such as the current scenario like mine where I don't expect to have production scale events.
Further, such cases (in general) can be part of some kind of cli argument where the user has an option to explicitly disable such recommended production settings which may not be required for a staging / experimentation.

Thanks for opening an issue. This is a known issue (should be documented).
Yes, unfortunately Lambda sets a very low default account concurrency. They will quickly increase your quota upon request.
Can you clarify what you mean by additional pricing? There is no cost to having your account concurrency increased and we are using reserved concurrency which is free (unlike provisioned concurrency which has a cost).
As far as disabling concurrency, we would have to be careful as we actually use reserved concurrency to ensure a specific function is single threaded to avoid concurrency issues with Glue and Iceberg. If concurrency was disabled, depending on the scale of data, it could result in errors since concurrent executions could result in too many concurrent updates on a Glue Table.

Thanks for getting back quick.

Can you clarify what you mean by additional pricing? There is no cost to having your account concurrency increased and we are using reserved concurrency which is free (unlike provisioned concurrency which has a cost).

Sorry for my ignorance. I typed it out of my head as I was thinking of provisioned concurrency  while the error was actually about reserved concurrency.
My issue / feature to disable does  not make much sense now as there is no additional pricing as you said for reserved concurrency.  Also can you let me know what would be the minimum number that is to be requested for increasing the reserved concurrency quota for deploying matano.
It would be also great if this is documented as part of installation steps.

No problem, they are easy to confuse :)
It will technically be sufficient to request 51 or 100, but you can probably just request 1000 since that's the default per the AWS docs (ref), new accounts just have a lowered concurrency.
I'll make sure to update the docs.

Thanks.
Also when you document , kindly include the below details that your referred link suggests to be put as part of the support case.
This will help others to use it as is (as a template request) and raise a case instead of figuring out justification by themselves for experimentation / initial deployment as these details are something that a user cannot determine before he / she even deploys and tests matano.
    Concurrency limit requested
    Anticipated average number of requests that your function will receive per second
    Anticipated highest number of requests that your function will receive per second
    Anticipated runtime duration
    Function memory size
    Invocation type (event or request-response)
    Event source
    Load test results that explain the reason why the quota increase is needed
    Any additional information that will help us better understand your use case

Thank you so much Ram for the bug report! Let's add @rams3sh for the contributions. cc @all-contributors


wrong nginx configuration when starting threebot

os:osx
nginx installed with brew install nginx
threebot start --local
2020-07-13 10:16:18.265 | DEBUG    | jumpscale.entry_points.threebot:test_privileged_ports_bind:38 - nginx: the configuration file /var/folders/75/br_kw5l577zfjlk20fgfqzmr0000gn/T/tmp2hoxuhcy syntax is ok
2020-07-13 10:16:18.266 | DEBUG    | jumpscale.entry_points.threebot:test_privileged_ports_bind:38 - 2020/07/13 10:16:18 [warn] 81643#0: 1024 worker_connections exceed open file resource limit: 256
2020-07-13 10:16:18.266 | DEBUG    | jumpscale.entry_points.threebot:test_privileged_ports_bind:38 - nginx: configuration file /var/folders/75/br_kw5l577zfjlk20fgfqzmr0000gn/T/tmp2hoxuhcy test is successful
2020-07-13 10:16:24.236 | INFO     | jumpscale.core.application.application:start:23 - Application gedis is started, process id: 81641
2020-07-13 10:16:24.247 | INFO     | jumpscale.core.application.application:start:23 - Application threebot_default is started, process id: 81641
2020-07-13 10:16:24.249 | INFO     | jumpscale.servers.threebot.threebot:start:605 - Configuring package auth
2020-07-13 10:16:25.441 | INFO     | jumpscale.servers.threebot.threebot:start:605 - Configuring package chatflows
2020-07-13 10:16:25.485 | INFO     | jumpscale.servers.threebot.threebot:start:605 - Configuring package admin
2020-07-13 10:16:25.791 | INFO     | jumpscale.servers.threebot.threebot:start:605 - Configuring package weblibs
2020-07-13 10:16:25.848 | INFO     | jumpscale.servers.threebot.threebot:start:605 - Configuring package tfgrid_solutions
2020-07-13 10:16:25.962 | INFO     | jumpscale.servers.threebot.threebot:_install_all:483 - Configuring package polls
2020-07-13 10:16:27.501 | ERROR    | jumpscale.sals.nginx.nginx:generate_certificates:176 - Generating certificate failed
Saving debug log to /Users/rob/sandbox/var/log/certbot/letsencrypt.log
Error while running nginx -c /Users/rob/sandbox/cfg/nginx/main/nginx.conf -t.

nginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /Users/rob/sandbox/cfg/nginx/main/./default_443/server.conf:2
nginx: configuration file /Users/rob/sandbox/cfg/nginx/main/nginx.conf test failed

The nginx plugin is not working; there may be problems with your existing configuration.
The error was: MisconfigurationError('Error while running nginx -c /Users/rob/sandbox/cfg/nginx/main/nginx.conf -t.\n\nnginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /Users/rob/sandbox/cfg/nginx/main/./default_443/server.conf:2\nnginx: configuration file /Users/rob/sandbox/cfg/nginx/main/nginx.conf test failed\n')

2020-07-13 10:16:28.623 | ERROR    | jumpscale.sals.nginx.nginx:generate_certificates:176 - Generating certificate failed
Saving debug log to /Users/rob/sandbox/var/log/certbot/letsencrypt.log
Error while running nginx -c /Users/rob/sandbox/cfg/nginx/main/nginx.conf -t.

nginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /Users/rob/sandbox/cfg/nginx/main/./default_443/server.conf:2
nginx: configuration file /Users/rob/sandbox/cfg/nginx/main/nginx.conf test failed

The nginx plugin is not working; there may be problems with your existing configuration.
The error was: MisconfigurationError('Error while running nginx -c /Users/rob/sandbox/cfg/nginx/main/nginx.conf -t.\n\nnginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /Users/rob/sandbox/cfg/nginx/main/./default_443/server.conf:2\nnginx: configuration file /Users/rob/sandbox/cfg/nginx/main/nginx.conf test failed\n')

2020-07-13 10:16:28.633 | INFO     | jumpscale.servers.threebot.threebot:start:617 - Reloading nginx
2020-07-13 10:16:28.687 | INFO     | jumpscale.servers.threebot.threebot:start:622 - Threebot is running at http://localhost:8080 and https://localhost:8443

ports should be fixed by: https://github.com/threefoldfoundation/tfta-trader/commit/05e6c8c53a1d239db1db575531ca1dc9fbd34df2

scenario on how to reproduce: it's caused by having a package defining ports e.g 80, 443 and running threebot with --local

i will verify this

Verified
commit_id 44c902e63830c1298e9bac47bc0f25e699022b7c


Source Code

When will you open the source code ?

@Marryli Code is now open sourced! Sorry it took so long. Let me know if you find bugs or have questions!


Add check for AD schema version

It would be helpful to add a check for the AD forest schema version as the minimum version supported for Hybrid Azure AD join is  69 (Server 2012 R2)

@mzmaili Is this the right area to have this request considered?


Petpipe msft

Ready for your review Andrew and Mark. Please complete PR if content is in good state.

Ah, I forgot to modify the pre-req.md to have a reference to this appendix file...


Set pyqt5 version to >=5.11.2,<5.12 in requirements/runtime.txt

Newer versions of pyqt5 (current version is 5.15.0) causes "core
dump" during tests:
...
wbia/algo/graph/demo.py s.Fatal Python error: Aborted

Current thread 0x00007f8d0c9de740 (most recent call first):
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/backends/backend_qt5.py", line 122 in _create_qApp
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/backends/backend_qt5.py", line 226 in __init__
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/backends/backend_qt5agg.py", line 21 in __init__
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 3224 in new_figure_manager_given_figure
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 3218 in new_figure_manager
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/pyplot.py", line 525 in figure
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/pyplot.py", line 578 in gcf
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/pyplot.py", line 935 in gca
  File "/virtualenv/env3/lib/python3.6/site-packages/matplotlib/pyplot.py", line 2656 in hist
  File "/wbia/wildbook-ia/wbia/algo/graph/demo.py", line 761 in show_score_probs
  File "<doctest:/wbia/wildbook-ia/wbia/algo/graph/demo.py::DummyVerif.show_score_probs:0>", line 6 in <module>
  File "/virtualenv/env3/lib/python3.6/site-packages/xdoctest/doctest_example.py", line 556 in run
  File "/virtualenv/env3/lib/python3.6/site-packages/xdoctest/plugin.py", line 172 in runtest
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/runner.py", line 135 in pytest_runtest_call
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/runner.py", line 217 in <lambda>
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/runner.py", line 244 in from_call
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/runner.py", line 217 in call_runtest_hook
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/runner.py", line 100 in runtestprotocol
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/main.py", line 272 in pytest_runtestloop
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/main.py", line 247 in _main
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/virtualenv/env3/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/virtualenv/env3/lib/python3.6/site-packages/_pytest/config/__init__.py", line 125 in main
  File "/virtualenv/env3/bin/pytest", line 8 in <module>
Aborted (core dumped)

Did you happen to find anyone else with this issue and/or file an issue?

@mmulich No I didn't really look into it, I just tried to find a version that makes it work...


Setting searchResultBackgroundColor does not seem to change the color

I can set the tokenView.backgroundColor and it works.
When I set the tokenView.searchResultBackgroundColor it does not seem to affect the color of the search results table that appears.
Is there something else I need to do to affect the color of the search table's background?

Matthew

You may have changed the background color correctly, but it is not visible because row colors are still default. Please try changing table row colors.


Issue : Unable to install medspacy

Hello MedSpacy Team,
When I try running pip install medspacy on my windows 11 arc64 based machine, I get error as
`Collecting pyrush
Using cached PyRuSH-<IP_ADDRESS>.tar.gz (44 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [65 lines of output]
G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
WARNING: The wheel package is not available.
WARNING: The wheel package is not available.
error: subprocess-exited-with-error
    python setup.py egg_info did not run successfully.
    exit code: 1

    [13 lines of output]
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "G:\Users\Manish\AppData\Local\Temp\pip-wheel-xeccbc4v\quicksectx_5f923c3b9a5a425fb733c088b3cf3f6b\setup.py", line 50, in <module>
        ext_modules=cythonize(extensions, language_level=3, language='c++'),
      File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\Cython\Build\Dependencies.py", line 970, in cythonize
        module_list, module_metadata = create_extension_list(
      File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\Cython\Build\Dependencies.py", line 816, in create_extension_list
        for file in nonempty(sorted(extended_iglob(filepattern)), "'%s' doesn't match any files" % filepattern):
      File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\Cython\Build\Dependencies.py", line 114, in nonempty
        raise ValueError(error_msg)
    ValueError: 'quicksect/quicksect.pyx' doesn't match any files
    Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
    [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: metadata-generation-failed

  Encountered error while generating package metadata.

  See above for output.

  note: This is an issue with the package mentioned above, not pip.
  hint: See above for details.
  Traceback (most recent call last):
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\installer.py", line 75, in fetch_build_egg
      subprocess.check_call(cmd)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['G:\\Users\\Manish\\AppData\\Local\\Programs\\Python\\Python39\\python.exe', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', 'G:\\Users\\Manish\\AppData\\Local\\Temp\\tmpf_3glxtu', '--quiet', 'quicksectx']' returned non-zero exit status 1.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "G:\Users\Manish\AppData\Local\Temp\pip-install-ji1n8ofx\pyrush_8c84e80b4101459d96a0faeed59b4933\setup.py", line 36, in <module>
      setup(
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\__init__.py", line 152, in setup
      _install_setup_requires(attrs)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\__init__.py", line 147, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\dist.py", line 806, in fetch_build_eggs
      resolved_dists = pkg_resources.working_set.resolve(
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\pkg_resources\__init__.py", line 766, in resolve
      dist = best[req.key] = env.best_match(
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\pkg_resources\__init__.py", line 1051, in best_match
      return self.obtain(req, installer)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\pkg_resources\__init__.py", line 1063, in obtain
      return installer(requirement)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\dist.py", line 877, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\installer.py", line 77, in fetch_build_egg
      raise DistutilsError(str(e)) from e
  distutils.errors.DistutilsError: Command '['G:\\Users\\Manish\\AppData\\Local\\Programs\\Python\\Python39\\python.exe', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', 'G:\\Users\\Manish\\AppData\\Local\\Temp\\tmpf_3glxtu', '--quiet', 'quicksectx']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.`
Anyway to solve this ?
When I do pip list -> I see medspacy, but when I import it, it says No module named pyrush. I tried installing it too, but couldn't do it.
Error for installatio of PyRush
`  error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [65 lines of output]
G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
WARNING: The wheel package is not available.
WARNING: The wheel package is not available.
error: subprocess-exited-with-error
    python setup.py egg_info did not run successfully.
    exit code: 1

    [13 lines of output]
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "G:\Users\Manish\AppData\Local\Temp\pip-wheel-zl4i9ikq\quicksectx_fb84dedd31154e0abfc06ac570961e49\setup.py", line 50, in <module>
        ext_modules=cythonize(extensions, language_level=3, language='c++'),
      File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\Cython\Build\Dependencies.py", line 970, in cythonize
        module_list, module_metadata = create_extension_list(
      File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\Cython\Build\Dependencies.py", line 816, in create_extension_list
        for file in nonempty(sorted(extended_iglob(filepattern)), "'%s' doesn't match any files" % filepattern):
      File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\Cython\Build\Dependencies.py", line 114, in nonempty
        raise ValueError(error_msg)
    ValueError: 'quicksect/quicksect.pyx' doesn't match any files
    Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
    [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: metadata-generation-failed

  Encountered error while generating package metadata.

  See above for output.

  note: This is an issue with the package mentioned above, not pip.
  hint: See above for details.
  Traceback (most recent call last):
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\installer.py", line 75, in fetch_build_egg
      subprocess.check_call(cmd)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['G:\\Users\\Manish\\AppData\\Local\\Programs\\Python\\Python39\\python.exe', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', 'G:\\Users\\Manish\\AppData\\Local\\Temp\\tmp6hzdhp5z', '--quiet', 'quicksectx']' returned non-zero exit status 1.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "G:\Users\Manish\AppData\Local\Temp\pip-install-4aq34s8b\pyrush_ea2f97c245a24a7ead4d269b4e2ea9e5\setup.py", line 36, in <module>
      setup(
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\__init__.py", line 152, in setup
      _install_setup_requires(attrs)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\__init__.py", line 147, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\dist.py", line 806, in fetch_build_eggs
      resolved_dists = pkg_resources.working_set.resolve(
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\pkg_resources\__init__.py", line 766, in resolve
      dist = best[req.key] = env.best_match(
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\pkg_resources\__init__.py", line 1051, in best_match
      return self.obtain(req, installer)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\pkg_resources\__init__.py", line 1063, in obtain
      return installer(requirement)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\dist.py", line 877, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "G:\Users\Manish\AppData\Local\Programs\Python\Python39\lib\site-packages\setuptools\installer.py", line 77, in fetch_build_egg
      raise DistutilsError(str(e)) from e
  distutils.errors.DistutilsError: Command '['G:\\Users\\Manish\\AppData\\Local\\Programs\\Python\\Python39\\python.exe', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', 'G:\\Users\\Manish\\AppData\\Local\\Temp\\tmp6hzdhp5z', '--quiet', 'quicksectx']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.`
Thanks

Hello!
I believe that the issue is that you are using Python 3.9. Some of our dependencies have not yet been updated to support 3.9. Could you try using an environment in 3.8 and see if that works?

Hello @abchapman93 , thanks for the quick followup.
I tried using py3.8 on my linux SSH server. Works like charm.
I'll try to do the same thing on my local window machine, till then take care :)


Fix CodeInline to plain text incorrect escape

Fixes #495
This PR fixes CodeInlineRenderer the same way as LiteralInlineRenderer by checking EnableHtmlEscape and call different methods accordingly.

Thanks!

Thanks!


Window Size and privacy.resistFingerprinting

If, in about:config, the key privacy.resistFingerprinting is set to True, the addon cannot set the popup window size neither it's position.
(probably because it can't get screen size...)
Position won't probably have a workaround, but what about window size?

@maverick74 there are three solutions:

When add-on loaded, disable resistFingerprinting =>  detect screen  => enable resistFingerprinting again.

  browser.privacy.websites.resistFingerprinting.get({})
  .then((got) => { // get resistFingerprinting setting
    const levelOfControls = [
     'controllable_by_this_extension',
     'controlled_by_this_extension'
    ];
    if (got.value === true && levelOfControls.includes(got.levelOfControl)) {
      browser.privacy.websites.resistFingerprinting.set({value: false}) // disable it
      .then(() => {
        screen = {
          width: window.screen.width,
          height: window.screen.height,
          top: window.screen.top,
          left: window.screen.left,
        }; // get screen top, left, width and height
        browser.privacy.websites.resistFingerprinting.set({value: true}) // enable it again
        .then(() => {});
      });
    } else {
      screen = {
        width: window.screen.width,
        height: window.screen.height,
        top: window.screen.top,
        left: window.screen.left,
      }; // get screen top, left, width and height
    }
  });

this solution need one more permission "privacy" when install add-on.

Add 4 option fields, user can input their screen width, height, top and left manually.
combine 1&2, Add 4 option fields and a button. when button click, ask privacy permission temporarily( see: optional_permissions and permissions.request() ).
detect screen and save them, then remove permission (see: permissions.remove())

Let me know if you have more ideas?

From what i understand, it would allow users to
1.either set the values manually or
2. to get them automatically (by granting privacy permission to the addon)
If i understood it right, then it seems option 3 would be the way to go! :)

@maverick74 Sorry for the late reply, according to the document, the privacy permission can't be optional.
BTW, the Number 1 is the only solution.
Any ideas on this issue?

Well... between "1" and not working, i surely prefer "1" :).
I assume that, to detect screen, it will disable>enable resistFingerprinting only for the first time it runs, right (or manually at user's request)?

@ettoolong window positioning does not work as it should.
I've have it set to place the window on bottom-right, but it places' it on top, somewhere near the center...

@ettoolong ?!?!?! Hum??? Did i just broke GitHub or something??? How the h*** did my last comment ended up above yours???

@maverick74 you can follow below steps:

open add-on option page and check the screen settings (width/height/left/top)
disable privacy.resistFingerprinting in about:config
click Auto detect button, that will update the screen settings
enable privacy.resistFingerprinting in about:config

nop... not working! :(
As soon as i enable privacy.resistFingerprinting, it places the popup window 10px from top and 912px from left (in a screen with 1920*1080)

@ettoolong anything new about this bug?

no, sorry for that. :(