TO DO: NI-DAQmx and Image Acquisition

Notes from my (likely last) meeting with Sarwagya 4/17
Sarwagya left a separate folder on the rig computer with NI-DAQmx enhancements. NI-DAQmx overall approach: create tasks to send signals, trigger them later, in parallel, AI/DO voltage channels in task
6/15 - started working on this list.
TO DO:

[x] Delete old code / comments, e.g. most of fillPDout
[x] LED frequency -- Sarwagya mentioned that the signal to turn on the LED can possibly cause flickering artifacts. Depending on the camera sampling frequency, I think this could manifest as low-frequency changes in intensity that may be easy to overlook. I will just stay aware of this concern moving forward.  We may be able to confirm in the image readout that intensity / amount of light is reasonable
[x] Address NI issue (on takeRLI call): https://github.com/john-judge/PhotoZ_upgrades/issues/3#issuecomment-862550507
[ ] Efficiency -- avoid needing to copy images in Camera::acquireImages. This means no longer pre-allocating a contiguous block of memory for images, but instead save the pointers that PDV produces when pdv_wait_image returns. Wouldn't be a correct approach, see PDV docs -- namely multibuf function
[ ] Rename the DAP variables, functions, and other names to DAQmx Not worth doing
[x] acquiOneRecord -- implement correct behavior of trialCount (if trialCount is 0, use pseudoOutput, typically pre-stimulation data. Based on OG PhotoZ, the original behavior enabled numSkippedTrials and pseudo recording -- no longer needed, so disabled.
[x] This last point is more related to Camera.cpp, but Sarwagya's opinion is that the parallelized camera readout + deinterleave logic should all be encapsulated within the Camera class.
[ ] Added Dave class with turbo SM code. Based on Chun's notes and the observed slowness/lack of sync, maybe 'superframing' (chunking 10 or 100 frames per readout) is necessary? In any case, maybe we've run into too many independent issues to justify trying to rewrite Chun's code
[ ] Connect image readout from MainController::acqui to GUI -- display the image data.  DataArray needs to match the dimensions of the Camera output. Note: this was temporarily manipulated in ArrayWindow.h:

#define DEFAULT_ARRAY_WIDTH 2048//256
#define DEFAULT_ARRAY_HEIGHT 40 //40

[ ] Including moving around quadrants.
[ ] NI-DAQmx errors from calls in DapController::acqui were causing the slow image acquisition apparently
[ ] capture FP analog input into buf in DapController.cpp
[ ] Deinterleave testing and debugging
[ ] Reorder deinterleave and CDS subtract -- for efficiency
[ ] Write unit tests once everything is working.

Issue: Sarwagya's transition of outputs and pseudoOutputs from fstream to uint8 array was not complete in the main branch. Need to check on the rig computer and hopefully his development is complete and intact there.

6/21/21 progress
Stepping through with debugger, these are working fine from record > acquiOneRecord:

arrangeData: stores input to raw_diode_data[i][trialNo][j]
getBinnedDiodeTrace loads a data slot (Data::rawData)
process > dataInversing > Data::raw2pro ends up putting in a constant factor in all data, as affected by CDS_add

Then I disabled CDS_subtract, deinterleave, and other camera-related processing. The arrays were still blank.
So in other words, the camera readout itself is now blank?
Next step: print out the camera readout directly to file, like previously.
Tried reverting to previous version of PhotoZ Little Dave; promptly BSOD-crashed.

##6/22/21 progress
Working version! reverted state of takeRLI
Data taken to reassemble and compare locally here
TurboSM image for validation:

6/23/21 Progress
LED goes on at Frame 200
The LED light should probably saturate the field of view

Ironed out a few bugs with data alignment. There's not much more validation I can do at this point. Next: investigate .cfg files and see if possible to manipulate image dimension -- try to discover how this is done in TurboSM.

When quadrant width was 1024px, reset rows were 1024px. Now with quadrant width 2048px, reset rows are still 1024px. Apparently, reset rows seem to be packed into the quadrant width, which might explain why all quadrant widths are multiples of 1024:
{ "2kx2k_NEURO_BINNED_d_", { 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024 },           //width
                           { 512, 200, 50, 160, 32, 15, 128, 40 } },                     //height

This also explains some of the seemingly mismatched dimensions. Take for example PhotoZ’s program #7 (which is TurboSM’s program #0):
4 quadrants * (2048 x 512)
Divide width by 2 for reset data:
1024x512
Multiply both height and width by 2 for quadrants:
2048x1024
Which matches up to the label for this setting:
200 Hz   2048x1024
Now, this formula doesn’t apply to all of the programs. Which is why I think there must be other hidden factors and settings that change from program to program. This may mean needing to parametrize the deinterleave and subtractCDS code to handle all the programs differently.
You can see why at this point I’m wondering if we’ve only scratched the surface, and that the better alternative is to attempt to use Chun’s code directly (which is what Chun has recommended).

6/28/21 Progress
Adjusted PhotoZ internal dimension expectations to match internal settings in .cfg files. The mismatch with the final dimensions is to be expected, I believe -- there may be more processing to do (just a simple crop, at best?)
Disabled deinterleave to test images with CDS subtraction only. Seems to be working reasonably, for all settings (analyzed 48 images total).
Put image-plotting script in repository  https://github.com/john-judge/PhotoZ_Image.git since interactive matplotlib mode is preferred to Colab. Preferably will add an environment.yml but can just use the one from ZDA_Explorer.
git clone https://github.com/john-judge/PhotoZ_Image.git

6/29/21 Progress
Repeating same testing as yesterday, but with deinterleave added back in. This will slow processing down.
If there's strange inexplicable behavior, I would remove CDS subtract and test deinterleave alone.

6/30/21 Progress
Developed the idea of the "missing width factor" -- for the 4 highest settings, there's no correction. For the lower settings, there's a divisive factor in just the width of 2 or 4. Perhaps cropping, or maybe another type of correction that affects only width? To search TurboSM.
In any case, there's something up with deinterleave -- none of the settings seem to work with the deinterleave code I developed to work with a single PDV channel readout. Would be great if I could install Anaconda/Jupyter notebook on the Little Dave rig computer -- would help guesswork efficiency.
const char* Camera::LABEL[] = {
"200 Hz   2048x1024", // = 2097152 px
"2000 Hz  2048x100",  // = 204800 px
"1000 Hz  1024x320",  // = 327680 px
"2000 Hz  1024x160",  // = 163840 px
"2000 Hz  512x160",   // = 81920 px
"4000 Hz  512x80",    // = 40960 px
"5000 Hz  256x60",    // = 15360 px
"7500 Hz  256x40" };  // = 10240 px

// The final widths to display
const int Camera::DISPLAY_WIDTH[] =  { 2048, 2048, 1024, 1024, 512, 512, 256, 256 };
const int Camera::DISPLAY_HEIGHT[] = { 1024, 100,  320,  160,  160, 80,  60,  40 };

// The internal quadrant widths based on .cfg files
const int Camera::WIDTH[] = { 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024 };
const int Camera::HEIGHT[] = { 512, 50,  160,   80,  80,   40,   30,  20 };

const int Camera::FREQ[] = { 200, 2000, 1000, 2000, 2000, 4000, 5000, 7500 };
const char* Camera::PROG[] = { //size size factor to display size
"DM2K_2048x512.cfg", // = 1048576 px 1x width 2x height
"DM2K_2048x50.cfg",  // = 102400 px 1x width 2x height
"DM2K_1024x160.cfg", // = 163840 px 1x width 2x height
"DM2K_1024x80.cfg",  // = 81920 px 1x width 2x height
"DM2K_1024x80.cfg",  // = 81920 px 1/2x width 2x height
"DM2K_1024x40.cfg",  // = 40960 px 1/2x width 2x height
"DM2K_1024x30.cfg",  // = 30720 px 1/4x width 2x height
"DM2K_1024x20.cfg"   // = 20480 px 1/4x width 2x height
};

// CDS subtraction => width is divided by a factor of 2
// Quadrants (4 PDV channels) => width and height are multiplied by a factor of 2
// Then all heights match final heights. However, there's a remaining width factor by which to divide:
const int Camera::WIDTH_DIVISIVE_FACTOR[] = { 1, 1, 1, 1, 2, 2, 4, 4 };

7/1/21 Progress
Testing with 2048x1024 (it's the most easily interpretable setting):

Tried stripes = 8, no change
Reverted stripes = 4 but kept channelOrder = twokchannel; num_pdvChan = 4; still no change
Realized there's a bug in pointers to images to process; corrected and reverted to original settings (stripes = 4). Different result, not easily troubleshootable
Reverted to my version of deinterleave
Changing channel order for ipdv=0 to 0,1,2,3
Disabled CDS-row deinterlace -- it doesn't seem correct based on result

channels to 0,1,2,3

Changing channel order for all PDV -- no effect
Changing up the width/height fed to deinterleave by factors of 2, preserving image size


STYLE-GUIDE: Suggest semantic line breaks

Per Slack discussion,
changes the recommendation on writing paragraphs
to suggest semantic line breaks instead.
The rationale behind the original recommendation was that
it made it easier to suggest changes to a pargraph during code review.
However, that's also possible with multi-line pargraphs
because GitHub supports multi-line comments and suggestions.
This new recommendation includes a new rationale for the choice
as well as a note about leaving existing documentation alone
until it's modified.

Rendered: https://github.com/pulumi/pulumi-hugo/blob/abhinav/sembr/STYLE-GUIDE.md#paragraphs-and-line-breaks

CC @iwahbe @AaronFriel


Return empty array if null body. Fixes gh-804

Codecov Report

Merging #806 into master will decrease coverage by <.01%.
The diff coverage is 66.66%.

@@             Coverage Diff              @@
##             master     #806      +/-   ##
============================================
- Coverage     50.59%   50.59%   -0.01%
  Complexity     2050     2050
============================================
  Files           261      261
  Lines          9860     9862       +2
  Branches       1547     1548       +1
============================================
+ Hits           4989     4990       +1
  Misses         4115     4115
- Partials        756      757       +1

Impacted Files
Coverage Δ
Complexity Δ

...tract/wiremock/restdocs/ContractResultHandler.java
34.72% <66.66%> (+0.21%)
9 <1> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9bafbf6...0785f40. Read the comment docs.


How can I follow a TRef?

Apologies for the naive question, I cannot seem to grasp how to follow a TRef to find the component it points to.
For example, I have a root file created with delphes (detector simulator). It has particles (MC truth objects), tracks and towers (observations from the simulated detector). The tracks and towers keep a TRef that points to the particle that created them. I want to follow this TRef and learn the index of the particle that made them. Getting the particle itself would probably be enough to figure out it's index (with some fuzzy comparisons).
The only method I can see on the TRef object is tref.read(source, cursor, context, parent) which has no doc-string available.
How should I follow a TRef in uproot?

Actually, I've never thought about how to follow a TRef, so in that sense, the feature does not exist in uproot. However, if you give me the file and point out where the TRef is that you're interested in, I'll see if it can easily be added. I'm sure the TRef contains a number, but the question is how to interpret that number. It would also help if you could tell me what that TRef is supposed to point to, which would help me figure out what relationship that number has with the object it references.

Wow, that would be very kind of you.
https://mega.nz/#F!afB2SSwC!3C8bp5xY_d01VvXmkh1nJQ
So there are two files on the other end of the link (hosted elsewhere because I cannot upload .root files here).
Inside the root file there are TRef at "Delphes"->"Track"->"Track.Particle"
These should point to parts of "Delphes"->"Particle", indicating which particle created each track
For example in event 0 the track 0 points to particle 751.
event_num | track_num | particle_num
0 | 0 | 751
0 | 1 | 765
0 | 2 | 779
0 | 3 | 782
0 | 4 | 797
The other file there is an SQLite database that I created from the root file... because I couldn't find anything that could directly read TRef's in python. ExRootAnalysis (part of Delphes) was used to make it in c++. The database may be a bit imperfect because it was constructed with a lot of fuzzy qualities, but it should be 99.9% accurate.

From your file, I implemented TRef as an object that can live in TTrees. The patch is in PR #326.
Here's an example of how to use them:
import uproot
t = uproot.open("issue324.root")["Delphes"]
refs = t["Track.Particle"].array()
refs
# <JaggedArray [
#     [<TRef 752> <TRef 766> <TRef 780> ... <TRef 1813> <TRef 1367> <TRef 1666>]
#     ...
#     [<TRef 745> <TRef 762> <TRef 783> ... <TRef 1863> <TRef 1713> <TRef 1717>]]>

These TRef objects each have an id:
refs[0][0].id
# 752

and following the normal rules of jagged arrays, you can ask for all of those ids in a single swipe:
refs.id
# <JaggedArray [
#      [752 766 780 ... 1813 1367 1666]
#      ...
#      [745 762 783 ... 1863 1713 1717]]>

Also following the normal rules of jagged arrays, you can pass them into a particle's attributes to pick the particles they refer to, except that these references start at 1 instead of starting at 0.
pt = t["Particle.PT"].array()
pt[refs.id - 1]
# <JaggedArray [
#      [0.7637838 1.1044897 5.463864 ... 4.252923 1.9702696 9.213475]
#      ...
#      [1.2523094 0.37887865 0.7390242 ... 1.0288503 3.4785874 1.804613]]>

We can see this by looking at the particles' fUniqueID:
t["Particle.fUniqueID"].array()
# <JaggedArray [
#      [1 2 3 ... 1811 1812 1813]
#      ...
#      [1 2 3 ... 1871 1872 1873]]>

and verify that they are strictly off-by-one (in each event, for all events):
(t["Particle.fUniqueID"].array() - 1 ==
 t["Particle.fUniqueID"].array().localindex
 ).all().all()
# True

I don't know if it is always true that the fUniqueIDs are strictly counting upward, starting at 1. In the general case, you'd have to inverse the lookup (i.e. "for which particle is the ID equal to REF?"), but as long as you're in this case, you can do forward indexing (i.e. "get the particle at position REF - 1").

This is now uproot 3.9.1.

Hi,
Sorry for necromancing but I just wanted to address your comment here, since I was stuck on it for a bit and it may be useful to someone else:

I don't know if it is always true that the fUniqueIDs are strictly counting upward, starting at 1. In the general case, you'd have to inverse the lookup (i.e. "for which particle is the ID equal to REF?"), but as long as you're in this case, you can do forward indexing (i.e. "get the particle at position REF - 1").

It is not always the case, in fact it's just a lucky coincidence that is specific to a particular configuration of Delphes. In general, TObjects are initialized with fUniqueID=-1. Any time TObject is added to TRefArray, it is assigned an fUniqueID, unless it already has a nonnegative one. Root maintains a static counter state via TProcessID and by default will increment the next fUniqueID every time a new one is issued. This is supposed to be unique at the process level, which is why a pid-dependent UID called "ProcessID0" is written to any root file containing a TRefArray.
Delphes overrides this behavior by resetting the counter at the beginning of every event it processes. It also as a global Factory object that preemptively assigns fUniqueIDs to any objects created that could potentially be written to file (whether or not they are ultimately referenced by a TRefArray or written out).
The lucky coincidence is that in the default configuration, usually the references to particles are references to the event-level collection Delphes/stableParticles, which happen to be the very first objects instantiated by the DelphesFactory, and are stored in the same order they are created.
However, if you were to, say, filter out some of these particles before writing them out, they fUniqueIDs would have no particular relationship with their TObjectArray indices in the file.
What's worse (and actually the genesis of issue #513), is that it's impossible to use this method to obtain a list of Tower objects from jets. Since Tower objects are created much later on during the event, they have large, random fUniqueID values, and something like Jet.Constituents will be a TRefArray containing those values. Instead, you need to go through the IDs in the TRefArray and pick out the corresponding objects from whatever array contains the object.

@cshimmin Thank you for writing this! Working on your issue got me thinking, "Maybe that's what those fUniqueIDs are for, after all: TRefs," and you confirmed it. I might need to formalize this connection with an Awkward IndexedArray, mapping TRef values to fUniqueID values, because otherwise I've been hiding the fUniqueIDs along with fBits. (Most of the time, users want this: half of the data in a TLorentzArray is in these two fields, but it only matters when those objects are being referred to by TRefs.
Or maybe it should be an option (keep_uniqueids)? Something like that.


Problems with multiple outputs and a single format

I'd like to output the results both in the console and in a file. I'd like to use progress formatter in both cases, but it does not work.
I tried with multiple format options
vendor/bin/behat --format progress --out std --format progress --out behat.log --strict

Nothing is displayed in the console and the results are doubled in the file:
...................................................................... 70
...................................................................... 140
...................................................................... 210
............................................

30 scenarios (30 passed)
254 steps (254 passed)
0m55.06s (155.63Mb)

30 scenarios (30 passed)
254 steps (254 passed)
0m55.06s (155.63Mb)

I have only 15 scenarios and 127 steps in my test suite
I tried with a single format option
vendor/bin/behat --format progress --out std --out behat.log --strict

The details are displayed in the console but the file is not created
I tried with a single format option and reordered the out options
vendor/bin/behat --format progress --out behat.log --out std --strict

Nothing appears in the console and the file is created.
I tried without format option
vendor/bin/behat --out behat.log --out std

The details are displayed in the console  but the file is not created
When I use different formats for the different outputs, it's working properly
vendor/bin/behat --format pretty --out behat.log --format progress --out std --strict

The details are displayed in the console and the file is created.

Only one output per format is currently supported.


Update domain layer schema document to v0.10.0

Documentation Scope

[x] update this domain layer schema doc
[x] convert the lark doc to a MD file and push it to the doc site.

Why
With this doc,

users can understand the algorithm of metrics in pre-built dashboards
users can do custom analysis based on the domain layer schema
contributors can understand the standard output when adding a new data source plugin

@yumengwang03 had finished the second task


⚠ Fetchers has degraded performance

In 1280b04, Fetchers ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 292 ms

Resolved: Fetchers performance has improved in aefec6e.


html element attribute name with hyphen will break in build/preview, but works in dev

Vue version
3.2.37
Link to minimal reproduction
https://sfc.vuejs.org/#eNp9Uk1PwzAM/Ssmlw5pbSR2QKo6NODCEQkkLrl0qzcKzYcSd4Cq/necrpvKkHaL/ez3nGd34t65bN+iyEURNr52BAGpdXfK1NpZT9CBxy30sPVWQ8KlyQl6tNqN+UzGIDIxDMpsrAkEOuxgGftnyRM2jYU365vqKrlWppAHORbigFC7piTkCKDYRN68TAm/aakEsygB8oCt/fiojWsJ9qm2FTbHKoYKeSITc3GYNNWlyz6CNfzNLnarEQhK5DBkYo6nj7ES70Qu5FK2xn3uMh5HrhiTvjVUa0wrq1eL7CZb3MqqDjTNZxh0uvb2K6BnRSXmE3LJyT361KOp0KO/KHZW+0fwDPsnGjV7ZXo24LiWuOCJy8VDS2RNdJ+K0Ni4B56Sug7KV7Ydeu5nl+WIFfLUMDF42N3Z2VS4rQ0+e+vCbHB24BtdVkQ/DnN4IV+b3egNd5RtwyVJvJ1h8OmBiP4X807pnQ==
Steps to reproduce

npm run build
npm run preview

What is expected?
successfully show UI
What is actually happening?
a "Uncaught type error: n.toUpperCase is not a function" throwed in console, and nothing is been displayed
System Info
WSL2 on windows:

  System:
    OS: Linux 5.4 Ubuntu 20.04.3 LTS (Focal Fossa)
    CPU: (20) x64 Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz
    Memory: 17.11 GB / 23.48 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 16.15.1 - ~/.nvm/versions/node/v16.15.1/bin/node
    Yarn: 1.22.19 - ~/.yarn/bin/yarn
    npm: 8.11.0 - ~/.nvm/versions/node/v16.15.1/bin/npm
  npmPackages:
    vue: ^3.2.37 => 3.2.37

Any additional comments?
If html element attribute name has a hyphen, it breaks after build->preview, but works in dev
<MyComponent a-text="blah" />

workaround: use camelCase would work in build and dev:
<MyComponent aText="blah" />

I had the same problem

it's not problem of Vue, but 3rd party library.  I couldn't remember how I fixed it, but you can start debugging from minimal Vue project then add your 3rd party libraries 1 by 1 to narrow down the library that caused problem.

I have been looking for two days, still have not found which third-party framework problems, hard to

Can you help me think about which framework, it seems to be related to the i18n problem, thank you very much 🙏

I can't remember.  It was pretty small fix.  If you can provide a mini repo that produce the bug, I can take a look.


Overcome two factor authentication

Describe the feature
I am trying to access the API using a cash account but it's not working and giving the login failed message while it's working perfectly fine with a paper account. Can you tell me how can I fix that or if possible can we add that authentication feature?
Expected interaction
Using a two-factor authentication device
Possible implications
Only cash account authentications
Additional context
Please suggest to me if there's any workaround to that

Thanks for the suggested enhancement 😊 Am I understanding correctly you're trying to figure out how to overcome the two-factor authentication with IBeam?

Thanks for the suggested enhancement 😊 Am I understanding correctly you're trying to figure out how to overcome the two-factor authentication with IBeam?

yeah, I tried all the way but I am still unable to use my cash account on the container. can you suggest to me how can I use my main account with that? Also, I am willing to put effort into it to make the two-factor auth possible somehow using the IBKR app. Please help me know if there's something I missed out on.

yeah, I tried all the way but I am still unable to use my cash account on the container. can you suggest to me how can I use my main account with that? Also, I am willing to put effort into it to make the two-factor auth possible somehow using the IBKR app. Please help me know if there's something I missed out on.

Right, I see. Unfortunately, IBeam does not handle 2FA automatically, therefore if this is what's stopping the authentication on your cash account then there is nothing that can currently be done by default. You'd need to complete the 2FA each time using your 2FA method or find another method to complete it automatically.
If however it isn't the 2FA then I'd suggest you check that you don't have any other places - such as TWS or web services -  where you use the cash account credentials.
Could you share the IBeam output log in case there's something there that could indicate the problem? Please run it with IBEAM_LOG_LEVEL environment variable set to DEBUG.

Right, I see. Unfortunately, IBeam does not handle 2FA automatically, therefore if this is what's stopping the authentication on your cash account then there is nothing that can currently be done by default. You'd need to complete the 2FA each time using your 2FA method or find another method to complete it automatically.
If however it isn't the 2FA then I'd suggest you check that you don't have any other places - such as TWS or web services -  where you use the cash account credentials.
Could you share the IBeam output log in case there's something there that could indicate the problem? Please run it with IBEAM_LOG_LEVEL environment variable set to DEBUG.

I have solved this problem, contact me to help you out.

I have solved this problem, contact me to help you out.

I have solved this problem, contact me to help you out.

that sounds great.
Can you please explain to me how can I do that?
thank you

I have solved this problem, contact me to help you out.

that sounds great.
Can you please explain to me how can I do that?
thank you

hey @jembl super cool you found a way around it! Could I ask you to share your solution here so others can use your findings in the future too? Thanks! 😊

hey @jembl super cool you found a way around it! Could I ask you to share your solution here so others can use your findings in the future too? Thanks! 😊

yes, so here is how I did it:

I got a www.twilio.com account and got a number, changed my IB phone number to twilio phone number.

I add a webhook for SMS, when an SMS is received, it will call my webserver i.e. www.sample.com/smshook

use this code to parse the SMS on the server side and upload it to a storage of my choice
`public String receive(String message) {
StringBuilder sb = new StringBuilder();
ResourceSet messages = Message.reader().limit(10).read();
 for(Message record : messages) {
     if (!smsStorageManager.getSmsWithSid(record.getSid()).isPresent()) {
         String messageSid = record.getSid();
         String messageBody = record.getBody();
         try {
             log.info("received message with SID " + record.getSid());
             log.info("received message with BODY " + record.getBody());
             String[] splits = messageBody.split(":");
             if (messageBody.startsWith("Your requested authentication code: ")) {
                 String code = splits[1].trim();
                 twoFactorCodeBlobStorageManager.insertIntoCodeBlob(code);
             }
             sb.append(messageSid + "\n");
         } catch (Exception e) {
             log.error("Failed to process the message " + messageBody, e);
         } finally {
             Message.deleter(messageSid).delete();
         }
     }
 }

 return sb.toString();

}`

On the container side i read the code from storage after login inside gateway_client.py:
`       sf_select_present = EC.presence_of_element_located((By.ID, _SF_SELECT_EL_ID))
WebDriverWait(driver, 15).until(sf_select_present)
_LOGGER.info("sf_select has appeared")
sf_select_el = Select(driver.find_element_by_id(_SF_SELECT_EL_ID))
sf_select_el.select_by_value('4.2')
_LOGGER.info("sf_select one time passcode")
  two_factor_input_present = EC.presence_of_element_located((By.ID, _TWO_FAC_EL_ID))
  WebDriverWait(driver, 15).until(two_factor_input_present)
  time.sleep(45)
  two_factor_el = driver.find_element_by_id(_TWO_FAC_EL_ID)
  _LOGGER.info(two_factor_el.text)
  two_factor_el.send_keys(get_latest_authentication_code())

  _LOGGER.info('Submitting the form for two fac')
  submit_form_el = driver.find_element_by_id(_SUBMIT_EL_ID)
  submit_form_el.click()

  success_present = EC.text_to_be_present_in_element((By.TAG_NAME, 'pre'), _SUCCESS_EL_TEXT)
  WebDriverWait(driver, _OAUTH_TIMEOUT).until(success_present)
  _LOGGER.info(driver.page_source.encode("utf-8"))

  _LOGGER.debug('Client login succeeds')`

and the code to get SMS from storage:
def get_latest_authentication_code():     connection_string = "YAYAYAYAYYA"     blob_client = BlobClient.from_connection_string(         connection_string, container_name="sms", blob_name="code.text")     # [START download_a_blob]     _LOGGER.info("downloading the code from blob")     download_stream = blob_client.download_blob()     code = download_stream.content_as_text()     _LOGGER.info("done downloading the code from blob ")     _LOGGER.info(str(code))     return code
Hope this helps, ask me if you have any questions!

yes, so here is how I did it:

I got a www.twilio.com account and got a number, changed my IB phone number to twilio phone number.

I add a webhook for SMS, when an SMS is received, it will call my webserver i.e. www.sample.com/smshook

use this code to parse the SMS on the server side and upload it to a storage of my choice
`public String receive(String message) {
StringBuilder sb = new StringBuilder();
ResourceSet messages = Message.reader().limit(10).read();
 for(Message record : messages) {
     if (!smsStorageManager.getSmsWithSid(record.getSid()).isPresent()) {
         String messageSid = record.getSid();
         String messageBody = record.getBody();
         try {
             log.info("received message with SID " + record.getSid());
             log.info("received message with BODY " + record.getBody());
             String[] splits = messageBody.split(":");
             if (messageBody.startsWith("Your requested authentication code: ")) {
                 String code = splits[1].trim();
                 twoFactorCodeBlobStorageManager.insertIntoCodeBlob(code);
             }
             sb.append(messageSid + "\n");
         } catch (Exception e) {
             log.error("Failed to process the message " + messageBody, e);
         } finally {
             Message.deleter(messageSid).delete();
         }
     }
 }

 return sb.toString();

}`

On the container side i read the code from storage after login inside gateway_client.py:
`       sf_select_present = EC.presence_of_element_located((By.ID, _SF_SELECT_EL_ID))
WebDriverWait(driver, 15).until(sf_select_present)
_LOGGER.info("sf_select has appeared")
sf_select_el = Select(driver.find_element_by_id(_SF_SELECT_EL_ID))
sf_select_el.select_by_value('4.2')
_LOGGER.info("sf_select one time passcode")
  two_factor_input_present = EC.presence_of_element_located((By.ID, _TWO_FAC_EL_ID))
  WebDriverWait(driver, 15).until(two_factor_input_present)
  time.sleep(45)
  two_factor_el = driver.find_element_by_id(_TWO_FAC_EL_ID)
  _LOGGER.info(two_factor_el.text)
  two_factor_el.send_keys(get_latest_authentication_code())

  _LOGGER.info('Submitting the form for two fac')
  submit_form_el = driver.find_element_by_id(_SUBMIT_EL_ID)
  submit_form_el.click()

  success_present = EC.text_to_be_present_in_element((By.TAG_NAME, 'pre'), _SUCCESS_EL_TEXT)
  WebDriverWait(driver, _OAUTH_TIMEOUT).until(success_present)
  _LOGGER.info(driver.page_source.encode("utf-8"))

  _LOGGER.debug('Client login succeeds')`

and the code to get SMS from storage:
def get_latest_authentication_code():     connection_string = "YAYAYAYAYYA"     blob_client = BlobClient.from_connection_string(         connection_string, container_name="sms", blob_name="code.text")     # [START download_a_blob]     _LOGGER.info("downloading the code from blob")     download_stream = blob_client.download_blob()     code = download_stream.content_as_text()     _LOGGER.info("done downloading the code from blob ")     _LOGGER.info(str(code))     return code
Hope this helps, ask me if you have any questions!

That is fantastic @jembl! 👏
Thank you so much for following up with such a detailed explanation! I was hypothesising that such an approach would be doable but I haven't looked that direction yet. Super happy to see this method worked for you and seriously congrats on devising it on your own!
Hey would you be interested in contributing some of these changes to IBeam? So that conditionally IBeam could be set to download the code from a URL (or some other way) and paste it to the login page. This way IBeam's users would only need to set up the SMS client and a pipeline to store the code. I could help you get this done, although no worries if you're busy or not interested, this answer is already a massive help 😊

That is fantastic @jembl! 👏
Thank you so much for following up with such a detailed explanation! I was hypothesising that such an approach would be doable but I haven't looked that direction yet. Super happy to see this method worked for you and seriously congrats on devising it on your own!
Hey would you be interested in contributing some of these changes to IBeam? So that conditionally IBeam could be set to download the code from a URL (or some other way) and paste it to the login page. This way IBeam's users would only need to set up the SMS client and a pipeline to store the code. I could help you get this done, although no worries if you're busy or not interested, this answer is already a massive help 😊

yes, so here is how I did it:

I got a www.twilio.com account and got a number, changed my IB phone number to twilio phone number.

I add a webhook for SMS, when an SMS is received, it will call my webserver i.e. www.sample.com/smshook

use this code to parse the SMS on the server side and upload it to a storage of my choice
`public String receive(String message) {
StringBuilder sb = new StringBuilder();
ResourceSet messages = Message.reader().limit(10).read();
 for(Message record : messages) {
     if (!smsStorageManager.getSmsWithSid(record.getSid()).isPresent()) {
         String messageSid = record.getSid();
         String messageBody = record.getBody();
         try {
             log.info("received message with SID " + record.getSid());
             log.info("received message with BODY " + record.getBody());
             String[] splits = messageBody.split(":");
             if (messageBody.startsWith("Your requested authentication code: ")) {
                 String code = splits[1].trim();
                 twoFactorCodeBlobStorageManager.insertIntoCodeBlob(code);
             }
             sb.append(messageSid + "\n");
         } catch (Exception e) {
             log.error("Failed to process the message " + messageBody, e);
         } finally {
             Message.deleter(messageSid).delete();
         }
     }
 }

 return sb.toString();

}`

On the container side i read the code from storage after login inside gateway_client.py:
`       sf_select_present = EC.presence_of_element_located((By.ID, _SF_SELECT_EL_ID))
WebDriverWait(driver, 15).until(sf_select_present)
_LOGGER.info("sf_select has appeared")
sf_select_el = Select(driver.find_element_by_id(_SF_SELECT_EL_ID))
sf_select_el.select_by_value('4.2')
_LOGGER.info("sf_select one time passcode")  two_factor_input_present = EC.presence_of_element_located((By.ID, _TWO_FAC_EL_ID))
  WebDriverWait(driver, 15).until(two_factor_input_present)
  time.sleep(45)
  two_factor_el = driver.find_element_by_id(_TWO_FAC_EL_ID)
  _LOGGER.info(two_factor_el.text)
  two_factor_el.send_keys(get_latest_authentication_code())

  _LOGGER.info('Submitting the form for two fac')
  submit_form_el = driver.find_element_by_id(_SUBMIT_EL_ID)
  submit_form_el.click()

  success_present = EC.text_to_be_present_in_element((By.TAG_NAME, 'pre'), _SUCCESS_EL_TEXT)
  WebDriverWait(driver, _OAUTH_TIMEOUT).until(success_present)
  _LOGGER.info(driver.page_source.encode("utf-8"))

  _LOGGER.debug('Client login succeeds')`

and the code to get SMS from storage:
def get_latest_authentication_code(): connection_string = "YAYAYAYAYYA" blob_client = BlobClient.from_connection_string( connection_string, container_name="sms", blob_name="code.text") # [START download_a_blob] _LOGGER.info("downloading the code from blob") download_stream = blob_client.download_blob() code = download_stream.content_as_text() _LOGGER.info("done downloading the code from blob ") _LOGGER.info(str(code)) return code
Hope this helps, ask me if you have any questions!

yes, so here is how I did it:

I got a www.twilio.com account and got a number, changed my IB phone number to twilio phone number.

I add a webhook for SMS, when an SMS is received, it will call my webserver i.e. www.sample.com/smshook

use this code to parse the SMS on the server side and upload it to a storage of my choice
`public String receive(String message) {
StringBuilder sb = new StringBuilder();
ResourceSet messages = Message.reader().limit(10).read();
 for(Message record : messages) {
     if (!smsStorageManager.getSmsWithSid(record.getSid()).isPresent()) {
         String messageSid = record.getSid();
         String messageBody = record.getBody();
         try {
             log.info("received message with SID " + record.getSid());
             log.info("received message with BODY " + record.getBody());
             String[] splits = messageBody.split(":");
             if (messageBody.startsWith("Your requested authentication code: ")) {
                 String code = splits[1].trim();
                 twoFactorCodeBlobStorageManager.insertIntoCodeBlob(code);
             }
             sb.append(messageSid + "\n");
         } catch (Exception e) {
             log.error("Failed to process the message " + messageBody, e);
         } finally {
             Message.deleter(messageSid).delete();
         }
     }
 }

 return sb.toString();

}`

On the container side i read the code from storage after login inside gateway_client.py:
`       sf_select_present = EC.presence_of_element_located((By.ID, _SF_SELECT_EL_ID))
WebDriverWait(driver, 15).until(sf_select_present)
_LOGGER.info("sf_select has appeared")
sf_select_el = Select(driver.find_element_by_id(_SF_SELECT_EL_ID))
sf_select_el.select_by_value('4.2')
_LOGGER.info("sf_select one time passcode")  two_factor_input_present = EC.presence_of_element_located((By.ID, _TWO_FAC_EL_ID))
  WebDriverWait(driver, 15).until(two_factor_input_present)
  time.sleep(45)
  two_factor_el = driver.find_element_by_id(_TWO_FAC_EL_ID)
  _LOGGER.info(two_factor_el.text)
  two_factor_el.send_keys(get_latest_authentication_code())

  _LOGGER.info('Submitting the form for two fac')
  submit_form_el = driver.find_element_by_id(_SUBMIT_EL_ID)
  submit_form_el.click()

  success_present = EC.text_to_be_present_in_element((By.TAG_NAME, 'pre'), _SUCCESS_EL_TEXT)
  WebDriverWait(driver, _OAUTH_TIMEOUT).until(success_present)
  _LOGGER.info(driver.page_source.encode("utf-8"))

  _LOGGER.debug('Client login succeeds')`

and the code to get SMS from storage:
def get_latest_authentication_code(): connection_string = "YAYAYAYAYYA" blob_client = BlobClient.from_connection_string( connection_string, container_name="sms", blob_name="code.text") # [START download_a_blob] _LOGGER.info("downloading the code from blob") download_stream = blob_client.download_blob() code = download_stream.content_as_text() _LOGGER.info("done downloading the code from blob ") _LOGGER.info(str(code)) return code
Hope this helps, ask me if you have any questions!

thank you so much @jembl
I understand what you are trying to implement here and got the Twilio part working now working on adding this code to the container. I am still in doubt on exactly at what part do I need to add these lines.
thank you

thank you so much @jembl
I understand what you are trying to implement here and got the Twilio part working now working on adding this code to the container. I am still in doubt on exactly at what part do I need to add these lines.
thank you

Hey @odssh and @jembl 👋 I've got some good news! There's a new version of IBeam currently in testing that should facilitate implementing this pattern. In this new version you will have an opportunity to set up IBeam in a way that will attempt to pull the 2FA code from an external URL.
You can try it out by using the image with tag 0.3.0-rc1, eg:
docker pull voyz/ibeam:0.3.0-rc1

Then when running it, do the following:

Set IBEAM_TWO_FA_HANDLER env var to EXTERNAL_REQUEST
Set IBEAM_EXTERNAL_REQUEST_URL env var to the URL you'd like to call
Set other env variables according to your requirements, see the external_request_handler.py

Alternatively, if you require full customisation in how you acquire the code you can do the following:

Provide the Inputs Directory with a script containing a class inheriting the TwoFaHandler and implementing the get_two_fa_code method - synchronously returning the 2FA code upon completion.
When running IBeam, set:

IBEAM_TWO_FA_HANDLER to CUSTOM_HANDLER
IBEAM_CUSTOM_TWO_FA_HANDLER to your_module_name.YourHandlerClass, corresponding to the script you provided in the Inputs Directory. Eg. for a file called my_handler.py and a class MyTwoFaHandler you'd set it as my_handler.MyTwoFaHandler.

Let me know if you'd have any feedback! 😊

Hi,
I did test that and put it to run. It ran fine for a few hours so I put it on run for two days and I just checked on it and I got about more than a thousand messages. Something is really wrong.
I am trying to identify it. Please let me know if you can suggest to me where it could break or any potential reason for that bug.
thank you

Hi @odssh thanks for testing it! Sorry to hear it started breaking after a while. Could you copy the output log over to here so I could give it a go at figuring out what's wrong? Thanks!

Hi,
Here's the log file for the whole session. It worked almost fine for the first day and then I got locked out of my account because of the excessive number of login trials as it kept making multiple login attempts each time it needed to log in.
In the log file, the line below is printed by me:
{"log":"Your requested authentication code: 433235\n","stream":"stdout","time":"2021-02-21T23:22:47.664576083Z"}
Please let me know if there's any fix for these multiple login attempts as it causes the account to get locked out from the Interactive broker's side.
thank you

That's fantastic @odssh thanks for passing this over and for the explanation! I'm really sorry you hit the lock out from IB - it happened to me too when I was testing this new method. I think it may be a good idea to implement some safety mechanism that will stop IBeam from attempting after a number of times as to prevent the lock-out from happening.
As for the output log, I'd have a number of questions:

Are you using IBeam as a Docker image, or standalone?

How are you printing this line? Your requested authentication code: 433235? This must come from some custom code, yet IBeam doesn't support that other than through CustomTwoFaHandler.

How is this message printed: Did not receive the authentication code? I cannot find it anywhere in IBeam. Have you modified it in some way? Could you let me see your fork if that's the case?

What IBEAM_TWO_FA_HANDLER are you using? The log seems to suggest that you're using the EXTERNAL_REQUEST TwoFaHandler, however I'm surprised to see that when it prints in the log it does not contain any URL field, ie. url=None:
2021-02-21 23:22:28,233|I| Attempting to acquire 2FA code from:  ExternalRequestTwoFaHandler(method=GET, url=None, timeout=300, params=None, headers=None)

If so, can you confirm that you're setting the URL correctly using IBEAM_EXTERNAL_REQUEST_URL?

It seems the 2FA code acquisition you perform is not acting fast enough. In multiple places in the output log I can see that IBeam requests the code, then waits and eventually times out. Only once that happens, your custom code seems to print the 2FA code. Could you share how your 2FA acquisition code looks like just to verify it isn't the issue?
(example of that behaviour):
2021-02-21T23:22:28.233255303Z -- 2021-02-21 23:22:28,232|I| Credentials correct, but Gateway requires two-factor authentication.
2021-02-21T23:22:28.233567968Z -- 2021-02-21 23:22:28,233|I| Attempting to acquire 2FA code from: ExternalRequestTwoFaHandler(   method=GET, url=None, timeout=300, params=None, headers=None)
2021-02-21T23:22:32.279821579Z -- 2021-02-21 23:22:32,279|E| Did not receive the authentication code
2021-02-21T23:22:32.280179965Z -- 2021-02-21 23:22:32,279|I| No 2FA code returned.
2021-02-21T23:22:47.55094453Z -- 2021-02-21 23:22:47,549|E| Error encountered during authentication
2021-02-21T23:22:47.550985101Z -- Traceback (most recent call last):
2021-02-21T23:22:47.55099196Z --   File \"/srv/ibeam/src/authenticate.py\", line 164, in authenticate_gateway
2021-02-21T23:22:47.551019941Z --     trigger = WebDriverWait(driver, var.OAUTH_TIMEOUT).until(any_of(success_present,    error_displayed))
2021-02-21T23:22:47.551026047Z --   File \"/opt/venv/lib/python3.7/site-packages/selenium/webdriver/support/wait.py\", line 80, in    until
2021-02-21T23:22:47.551031437Z --     raise TimeoutException(message, screen, stacktrace)
2021-02-21T23:22:47.551036576Z -- selenium.common.exceptions.TimeoutException: Message:
2021-02-21T23:22:47.551041561Z --
2021-02-21T23:22:47.551046234Z --
2021-02-21T23:22:47.551050865Z -- The above exception was the direct cause of the following exception:
2021-02-21T23:22:47.551055927Z --
2021-02-21T23:22:47.551060554Z -- Traceback (most recent call last):
2021-02-21T23:22:47.551065397Z --   File \"/srv/ibeam/src/authenticate.py\", line 176, in authenticate_gateway
2021-02-21T23:22:47.551070461Z --     raise RuntimeError('Error encountered during authentication') from e
2021-02-21T23:22:47.551075534Z -- RuntimeError: Error encountered during authentication
2021-02-21T23:22:47.664339484Z -- 2021-02-21 23:22:47,663|I| Authentication process failed
2021-02-21T23:22:47.664509274Z -- Your requested authentication code: 433235

Note how the authentication fails due to no code being returned at 23:22:32, while your message is printed at 23:22:47, over 10 seconds later. It seems that the code is received a little too late. It would be great if we could figure out why that hold happens.

If you could feed me in on as many details regarding these questions as possible, I should be able to understand what is causing the issue on your end. Appreciate your help in the matter!

thank you for your quick response @Voyz.
Here's the python file that I have modified. In this file, I have changed only lines 12, 13, and 60-118. I know I should have gone with two_fa_handler.py but at the moment external_request_handler.py seemed easier.
Here are the answers to your question:

Docker
line 116
line 85
EXTERNAL_REQUEST
the code in the file attached checks for verification messages constantly and returns it as soon as it gets it.
Right now I am not sure what causing the issue.
Also, one more thing about the docker(I am just getting started), why am I able to request localhost 5000 inside the container but not outside even though I added a -p tag with port while launching the container. It simply says "Access denied".
thank you

Just a quick message - I have edited your last reply as the link you provided contains your authentication credentials. Below is your code with the credentials removed. I suggest you remove this gist as soon as possible as to not leak your credentials.
import json
import logging
import os
from pathlib import Path
from typing import Union

import requests

from ibeam.src.two_fa_handlers.two_fa_handler import TwoFaHandler

from datetime import datetime, timezone
from twilio.rest import Client

_LOGGER = logging.getLogger('ibeam.' + Path(__file__).stem)

_EXTERNAL_REQUEST_METHOD = os.environ.get('IBEAM_EXTERNAL_REQUEST_METHOD', 'GET')
"""Method to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_URL = os.environ.get('IBEAM_EXTERNAL_REQUEST_URL')
"""URL to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_TIMEOUT = int(os.environ.get('IBEAM_EXTERNAL_REQUEST_TIMEOUT', 300))
"""URL to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_PARAMS = os.environ.get('IBEAM_EXTERNAL_REQUEST_PARAMS')
"""Params to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_HEADERS = os.environ.get('IBEAM_EXTERNAL_REQUEST_HEADERS')
"""Headers to use by the external request 2FA handler."""

def parse_json(s):
    if s is None:
        return None
    try:
        return json.loads(s)
    except Exception as e:
        _LOGGER.exception(e)
        return None

class ExternalRequestTwoFaHandler(TwoFaHandler):

    def __init__(self, method: str = None,
                 url: str = None,
                 timeout: int = None,
                 params=None,
                 headers=None):

        self.method = method if method is not None else _EXTERNAL_REQUEST_METHOD
        self.url = url if url is not None else _EXTERNAL_REQUEST_URL
        self.timeout = timeout if timeout is not None else _EXTERNAL_REQUEST_TIMEOUT
        self.params = params if params is not None else parse_json(_EXTERNAL_REQUEST_PARAMS)
        self.headers = headers if headers is not None else parse_json(_EXTERNAL_REQUEST_HEADERS)

    def get_two_fa_code(self) -> Union[str, None]:
        try:
            # response = requests.request(method=self.method,
            #                             url=self.url,
            #                             timeout=self.timeout,
            #                             params=self.params,
            #                             headers=self.headers, )
            # response.raise_for_status()
            # return response.content
            return self.twilioAuthToken()

        except requests.exceptions.HTTPError as err:
            _LOGGER.error(err)
            return None

    def twilioAuthToken(self):
        vCode = None
        for i in range(20):
            vCode = self.getNewMessage()
            if vCode != None:
                break
            else:
                pass
        if vCode == None:
            _LOGGER.error("Did not receive the authentication code")
        print("got it", vCode)

        return vCode

    def getNewMessage(self):
        vcode = None
        account_sid = 'ACCOUNT_SID'
        auth_token = 'AUTH_TOKEN'
        dtf = '%Y-%m-%d %H:%M:%S'
        client = Client(account_sid, auth_token)

        messages = client.messages.list(
            limit=1,
            to='+19876543210'
        )
        for record in messages:
            message = client.messages(record.sid).fetch()

            currentTime = datetime.now().timestamp() # current time in utc
            dts = str(message.date_created).split('+')[0]
            dtf = '%Y-%m-%d %H:%M:%S'
            dtSend = datetime.strptime(dts, dtf).replace(tzinfo = timezone.utc).timestamp() # message send time in utc

            if currentTime - dtSend <= 10:
                vcode2 = message.body
                try:
                    vcode1 = vcode2.split(':')[1]
                    vcode = int(vcode1.strip(' '))
                except:
                    vcode = None
            print(message.body)

        return vcode

    def __str__(self):
        return f"ExternalRequestTwoFaHandler(method={self.method}, url={self.url}, timeout={self.timeout}, params={self.params}, headers={self.headers})"

Now, thanks for your answers! That's all very helpful to know 😊

Also, one more thing about the docker(I am just getting started), why am I able to request localhost 5000 inside the container but not outside even though I added a -p tag with port while launching the container. It simply says "Access denied".

This is due to Gateway's config.yml not having allowed IPs set up. Have a look at Troubleshooting to learn about this issue and how to solve it - let me know if you'd have any troubles doing it and I'll be happy to help you.

Digging into the code I can recognise a few areas that may cause the issues, leading to the unexpected behaviour we're observing in your logs:

Firstly, you run 20 attempts to acquire the code:
for i in range(20):
    vCode = self.getNewMessage()

That 20 seems like an arbitrary number. There also isn't any time delay between these attempts, so what seems to be happening is that all 20 happen at roughly the same time. The logs seem to confirm this, as 20 prints from line 116 appear within milliseconds of each other. I would suggest that you base that loop on a timeout, rather than on an arbitrary number, introducing a time delay between attempts, eg.:
 def twilioAuthToken(self):
     vCode = None
     start_time = time.time()
     end_time = start_time + _TWILIO_TIMEOUT
     while time.time() < end_time:
         vCode = self.getNewMessage()
         if vCode != None:
             break
         time.sleep(1) # 1 second delay between requests

         _LOGGER.info(f"No code received, retrying for another {round(end_time - time.time())} seconds")

     if vCode == None:
         _LOGGER.error(f"Did not receive the authentication code after {round(time.time()-start_time)} seconds.")
     else:
         _LOGGER.info(f'Received code: {vCode}')

     return vCode

I also increased verbosity of your logs to get more information. I suggest you set IBEAM_LOG_LEVEL env variable to 'DEBUG' in order to see even more information in your logs during this development stage.

I cannot comment much on the way you're interacting with Twilio as I haven't used them - I'd double check that you're doing everything correctly there too. However, you seem to only look up the messages from the previous 10 seconds if I'm not mistaken:

```python
if currentTime - dtSend <= 10:
```
I'd suggest to try to increase that buffer, as this may simply not be enough for a SMS message to deliver.

Furthermore, I'd suggest to alter the way you parse the messages received:

if currentTime - dtSend <= 10:
      vcode2 = message.body
      if 'Your requested authentication code' not in vcode2 or not vcode2[-6:].isdigit():
          _LOGGER.error(f'Unrecognised message body: {vcode2}')
      else:

          try:
              vcode1 = vcode2.split(':')[1]
              vcode = int(vcode1.strip(' '))
          except Exception as e:
              _LOGGER.exception(e)
              vcode = None

This way we can be more sure that the right messages are being parsed, and that we don't return invalid codes (eg. in your existing code a message that reads Error : Couldn't access code would return Couldn't access code as the 2FA code)
Lastly, I suggest you remove the following lines:
 def __str__(self):
        return f"ExternalRequestTwoFaHandler(method={self.method}, url={self.url}, timeout={self.timeout}, params={self.params}, headers={self.headers})"

As they are misleading, and replace them with:
 def __str__(self):
        return f"TwilioHandler()"

I hope these changes help you ensure the right code is returned. We could certainly introduce some changes on IBeam to make things more robust in case wrong code is returned too many times - I'll try to look into this soon. Let me know how it goes! 👍

Just a quick message - I have edited your last reply as the link you provided contains your authentication credentials. Below is your code with the credentials removed. I suggest you remove this gist as soon as possible as to not leak your credentials.
import json
import logging
import os
from pathlib import Path
from typing import Union

import requests

from ibeam.src.two_fa_handlers.two_fa_handler import TwoFaHandler

from datetime import datetime, timezone
from twilio.rest import Client

_LOGGER = logging.getLogger('ibeam.' + Path(__file__).stem)

_EXTERNAL_REQUEST_METHOD = os.environ.get('IBEAM_EXTERNAL_REQUEST_METHOD', 'GET')
"""Method to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_URL = os.environ.get('IBEAM_EXTERNAL_REQUEST_URL')
"""URL to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_TIMEOUT = int(os.environ.get('IBEAM_EXTERNAL_REQUEST_TIMEOUT', 300))
"""URL to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_PARAMS = os.environ.get('IBEAM_EXTERNAL_REQUEST_PARAMS')
"""Params to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_HEADERS = os.environ.get('IBEAM_EXTERNAL_REQUEST_HEADERS')
"""Headers to use by the external request 2FA handler."""

def parse_json(s):
    if s is None:
        return None
    try:
        return json.loads(s)
    except Exception as e:
        _LOGGER.exception(e)
        return None

class ExternalRequestTwoFaHandler(TwoFaHandler):

    def __init__(self, method: str = None,
                 url: str = None,
                 timeout: int = None,
                 params=None,
                 headers=None):

        self.method = method if method is not None else _EXTERNAL_REQUEST_METHOD
        self.url = url if url is not None else _EXTERNAL_REQUEST_URL
        self.timeout = timeout if timeout is not None else _EXTERNAL_REQUEST_TIMEOUT
        self.params = params if params is not None else parse_json(_EXTERNAL_REQUEST_PARAMS)
        self.headers = headers if headers is not None else parse_json(_EXTERNAL_REQUEST_HEADERS)

    def get_two_fa_code(self) -> Union[str, None]:
        try:
            # response = requests.request(method=self.method,
            #                             url=self.url,
            #                             timeout=self.timeout,
            #                             params=self.params,
            #                             headers=self.headers, )
            # response.raise_for_status()
            # return response.content
            return self.twilioAuthToken()

        except requests.exceptions.HTTPError as err:
            _LOGGER.error(err)
            return None

    def twilioAuthToken(self):
        vCode = None
        for i in range(20):
            vCode = self.getNewMessage()
            if vCode != None:
                break
            else:
                pass
        if vCode == None:
            _LOGGER.error("Did not receive the authentication code")
        print("got it", vCode)

        return vCode

    def getNewMessage(self):
        vcode = None
        account_sid = 'ACCOUNT_SID'
        auth_token = 'AUTH_TOKEN'
        dtf = '%Y-%m-%d %H:%M:%S'
        client = Client(account_sid, auth_token)

        messages = client.messages.list(
            limit=1,
            to='PHONE_NUMBER'
        )
        for record in messages:
            message = client.messages(record.sid).fetch()

            currentTime = datetime.now().timestamp() # current time in utc
            dts = str(message.date_created).split('+')[0]
            dtf = '%Y-%m-%d %H:%M:%S'
            dtSend = datetime.strptime(dts, dtf).replace(tzinfo = timezone.utc).timestamp() # message send time in utc

            if currentTime - dtSend <= 10:
                vcode2 = message.body
                try:
                    vcode1 = vcode2.split(':')[1]
                    vcode = int(vcode1.strip(' '))
                except:
                    vcode = None
            print(message.body)

        return vcode

    def __str__(self):
        return f"ExternalRequestTwoFaHandler(method={self.method}, url={self.url}, timeout={self.timeout}, params={self.params}, headers={self.headers})"

I am sorry for the confusion. Actually, the credentials were not real and I just put them there as a placeholder. Also thank you for informing me about that.

Now, thanks for your answers! That's all very helpful to know 😊

Also, one more thing about the docker(I am just getting started), why am I able to request localhost 5000 inside the container but not outside even though I added a -p tag with port while launching the container. It simply says "Access denied".

This is due to Gateway's config.yml not having allowed IPs set up. Have a look at Troubleshooting to learn about this issue and how to solve it - let me know if you'd have any troubles doing it and I'll be happy to help you.
Digging into the code I can recognise a few areas that may cause the issues, leading to the unexpected behaviour we're observing in your logs:

Firstly, you run 20 attempts to acquire the code:
for i in range(20):
    vCode = self.getNewMessage()

That 20 seems like an arbitrary number. There also isn't any time delay between these attempts, so what seems to be happening is that all 20 happen at roughly the same time. The logs seem to confirm this, as 20 prints from line 116 appear within milliseconds of each other. I would suggest that you base that loop on a timeout, rather than on an arbitrary number, introducing a time delay between attempts, eg.:
 def twilioAuthToken(self):
     vCode = None
     start_time = time.time()
     end_time = start_time + _TWILIO_TIMEOUT
     while time.time() < end_time:
         vCode = self.getNewMessage()
         if vCode != None:
             break
         time.sleep(1) # 1 second delay between requests

         _LOGGER.info(f"No code received, retrying for another {round(end_time - time.time())} seconds")

     if vCode == None:
         _LOGGER.error(f"Did not receive the authentication code after {round(time.time()-start_time)} seconds.")
     else:
         _LOGGER.info(f'Received code: {vCode}')

     return vCode

I also increased verbosity of your logs to get more information. I suggest you set IBEAM_LOG_LEVEL env variable to 'DEBUG' in order to see even more information in your logs during this development stage.

I cannot comment much on the way you're interacting with Twilio as I haven't used them - I'd double check that you're doing everything correctly there too. However, you seem to only look up the messages from the previous 10 seconds if I'm not mistaken:
python     if currentTime - dtSend <= 10:
I'd suggest to try to increase that buffer, as this may simply not be enough for a SMS message to deliver.

Furthermore, I'd suggest to alter the way you parse the messages received:

if currentTime - dtSend <= 10:
      vcode2 = message.body
      if 'Your requested authentication code' not in vcode2 or not vcode2[-6:].isdigit():
          _LOGGER.error(f'Unrecognised message body: {vcode2}')
      else:

          try:
              vcode1 = vcode2.split(':')[1]
              vcode = int(vcode1.strip(' '))
          except Exception as e:
              _LOGGER.exception(e)
              vcode = None

This way we can be more sure that the right messages are being parsed, and that we don't return invalid codes (eg. in your existing code a message that reads Error : Couldn't access code would return Couldn't access code as the 2FA code)
Lastly, I suggest you remove the following lines:
 def __str__(self):
        return f"ExternalRequestTwoFaHandler(method={self.method}, url={self.url}, timeout={self.timeout}, params={self.params}, headers={self.headers})"

As they are misleading, and replace them with:
 def __str__(self):
        return f"TwilioHandler()"

I hope these changes help you ensure the right code is returned. We could certainly introduce some changes on IBeam to make things more robust in case wrong code is returned too many times - I'll try to look into this soon. Let me know how it goes! 👍

Thank you so much @Voyz
I think it will definitely solve the issue as mow I am realizing that it might be the loop of random number 20.
I will put the code on run and will let you know what I get.
About the point
2. 10 seconds was the time to check how old the message is so if the most recent message is older than 10 seconds then the script will keep waiting until a new message appears and that I did with a loop of 20 and I think that was causing the issue.
thank you so much for the time. It really helps a lot and will save a lot of time.
I will let you know how it goes.
thank you

Actually, the credentials were not real and I just put them there as a placeholder. Also thank you for informing me about that.

Ah glad to hear that! Sorry I stepped over then! 😅

10 seconds was the time to check how old the message is so if the most recent message is older than 10 seconds then the script will keep waiting until a new message appears and that I did with a loop of 20 and I think that was causing the issue.

Right, this may be a tricky one to figure out. It would be helpful to mark messages as 'read' and 'unread' on Twilio side - have a look if this is possible. A rigid 10 seconds limit may be a cause of bugs in the future.

I will let you know how it goes.

Great, looking forward to it! 😊

@odssh I've just released a new version -rc2, ie.: voyz/ibeam:0.3.0-rc2. It contains additional security measures, checking that the 2FA code provided is valid (ie. contains 6 digits). It can be disabled by setting IBEAM_STRICT_TWO_FA_CODE to False, and it is based on the assumption that all 2FA codes are 6-digit. Let me know if this is not the case for you. Additionally, the new version does a better job at handling cases where there is no 2FA code provided by the handler.
Looking forward to hearing how the improvements worked out for you 👍

Thank you for the detailed update @Voyz
I think the verification codes are 6 digit all the time. I will definitely help me a lot.
I am still working on previous fixes.
Thank you 👍

Hi @Voyz
I was running the container for the last two days and it broke yesterday.
As you can see in this log file was authenticated for a long time but then all of a sudden it started authenticating every single minute and was able to authenticate but still kept doing it until it got locked out and then it still kept trying. I tried to search for the potential cause but unable to find it.
Please let me know if there's any way to fix that.
thank you

hey @odssh I'm really sorry to hear about the second lock-out, that must be incredibly annoying and I'm going to try my best to help you avoid it in the future. Thanks for submitting the detailed log too. From that I can conclude the following:

IBeam worked fine for you from 2021-03-03 22:43:16 till 2021-03-04 22:01:33
During that time, it encountered some authentication drops and handled all of them well, except for one which returned 500 Internal Server Error. I haven't seen this error at all during development and I'd understand it as Gateway having some internal issues. Given that this happened only once and that authentication completed for another few hours after that I'd assume that it was not the cause for the later issues.
At 2021-03-04 22:01:33 Gateway started losing authentication and IBeam couldn't complete authentication since then.
From 2021-03-04 22:02:33 till 2021-03-04 23:22:39 - so for another 80 authentication attempts - IBeam was successfully retrieving and inputting the 2FA code along with the credentials. I conclude that by seeing the Authentication process succeeded message printed a lot in this time. At the same time, the error message Error displayed by the login webpage: failed appeared 9 times in that period. After the 9th time that error message appeared your account got locked out, although bear in mind this error also appeared earlier in the log once. We could assume 10 unsuccessful 2FA authentications cause a lock-out.
There are two problems at hand here:

These 10 failed attempts would signify that the 2FA code returned is not correct. If I remember correctly, your 2FA auth logic tries to pull the most recent messages and then uses the most recent one. Could there be a flaw in the logic somewhere there? I'd suggest you double check for every of these 10 error 2FA attempts that IBeam received the correct 2FA code. Look at the timestamps of the messages received and double check if they correspond to the times IBeam would have requested these - and see if indeed the right codes were returned to IBeam by looking at the Received code: logs. If there's a discrepancy here we at least have one culprit. If you could share the messages log from that period of time it would be useful too.
Despite the majority of successful attempts, Gateway didn't seem to be authenticated, as from that point every maintenance recognised an active unauthenticated session.

This second point is probably the core of the issue here. Why was the session not authenticated despite so many successful attempts? We can try to answer that by looking at the following:

Have you used these credentials to authenticate manually at that time? For instance to TWS, client portal or mobile app? If you could double check that, it would verify that lead.
Assuming you're running IBeam as a Docker image, is there any way your orchestration (or other system) would start two instances of IBeam? Could you find a way to confirm that only one IBeam instance was running at the time?
If both of the above leads don't provide any indications, we may try to read the Gateway logs. You may access these by looking in the clientportal.gw\logs directory on the container where IBeam is running. There you'll find two files gw.[DATE].log and gw.message.[DATE].log. If you can, either read through these logs from 2021-03-04 22:02:33 and try to understand what changed, or clean these files of any private information (some credentials are present there) and send them over in a private email to<EMAIL_ADDRESS>for me to look through these.

These steps should move us forward towards understanding what is wrong. Once again, I'm really sorry you've run into these issues and I'm hoping we'll mange to get to the bottom of this for the benefit of all users of IBeam. Thanks for your patience in the meantime 👍

Hi @Voyz
I am trying to build the docker image using the command below and the dockerfile from this repo:
docker build -t ib_auth_api .
And creating the container using the command below:
docker run --env-file creds.list -d -p 5000:5000 ib_auth_api
So I am able to access my account by open a bash shell inside the container but outside the container, it gives me the error saying access denied. I have also added all the additional files related to the gateway and all other needed. Can you please tell me how can I build my own image using your dockerfile. If possible can you share the command you used to build the image and the files that you have modified in the gateway directory. I have wasted more than a month on it and still unable to get it to work.
Thank you

hey @odssh - sorry to hear you're having so many troubles.
The command I use to build the image is this:
docker build -t ibeam .

The Docker commands you use seem fine to me.
I've recently realised that the original conf.yaml was indeed modified in IBeam - have a look at https://github.com/Voyz/ibeam/wiki/Gateway-Configuration to see the version IBeam uses. You may need to add 172.17.0.* to allowed IPs to communicate with IBeam from outside of Docker if using 'localhost' as the hostname. Have a look at #15 for more info on this. Also, check https://github.com/Voyz/ibeam/wiki/Troubleshooting#access-denied for more info on this error. Let me know if these help you with the Access Denied - otherwise try to describe exactly what you're doing and what commands you're using in as much detail as possible.
Also, could you outline why are you building the Docker image yourself instead of using voyz/ibeam:latest?

Thank you so much @Voyz
It finally worked. I wish I had asked it a month ago but I thought It was more of a docker related issue.
Again thank you for the quick reply.
Hope you have a wonderful day

Yay! @odssh Happy to hear! It was the 172.17.0.*? Yeah we only just figured it out recently, although you surely should feel free to report issues as soon as you encounter them, as possibly some of these are bugs and can be fixed relatively easily.
Can I nonetheless ask you to answer this question:

Also, could you outline why are you building the Docker image yourself instead of using voyz/ibeam:latest?

Just trying to understand whether there are some features you consider crucial that we could implement into IBeam by default, or some other issues that prevent you from using that image. Appreciate it 👍

I added both <IP_ADDRESS> and 172.17.0 and it seems to be working. I will test out both cases in detail later on.
I am still learning the docker so I just love to do it and try to practice as much as I can. Also since I have added Twilio code so It feels a lot more familiar for me to edit.
I will be more than happy to contribute to the project as soon as I will get good at it.
Thank you so much for your time.

Fantastic, thanks for your explanation 👍 It indeed is a great exercise to practice building someone else's code, and I'm both happy that you finally succeeded and sorry that you encountered so many troubles on your way. Other users also made their own forks of IBeam and shared it with us to the benefit of community, so if you'd be interested I'd highly encourage you to share your code with us and see what improvements you made that could benefit us all.
At this point I'd also highlight that in #12 we're implementing a standard way of handling 2FA, which I think may suit your Twilio use case. In fact, it would be an incredible test case if you managed to get Twilio to run 2FA with the version introduced in #12. It also contains many other improvements. If you'd be interested to test it out do let me know and I'll be happy to help you transition, although no worries if not - that version will be published soon nonetheless so you'll have an opportunity to try it out then.
All the best 👋 Voy

I will definitely share the changes I have made to the repo once I can get it all working. Since I have got the authenticator working so I am now moving on to my actual trader and just want to get it running somehow and then I will solely focus on IBeam authenticator to improve it as much as I can. Soon after I get my bot running, I will be working on adding the scheduler to the authenticator so that we only try to authenticate when we need to be authenticated. That feature would be great as an option as it can make the authenticator a lot more precise and efficient for the specific use cases.
I highly appreciate the time and support you are giving and will try my best to make as much contribution as I can to this project.
thank you

hey @odssh thanks for outlining all of that! I'd be interested to see and hear what scheduler for the authenticator would do. Keep us posted on your progress 👍
And thanks for the kind words, I'm happy to be able to support you in your trading 😊

thank you for all the help and I will definitely make sure to update you on that scheduler thing.
I have run into another issue. So while trying to add a proprietary certificate I got everything done but when I am running the container by mounting the inputs as a volume using the command below:
docker run -v /inputs:/srv/inputs --env-file paper.list -d -p 5000:5000 ib_with_ssl

I am getting the error on making requests. so when I ran the command
curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem

I am getting this error:
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
I followed the exact guide in the page (tls)[https://github.com/Voyz/ibeam/wiki/TLS-Certificates-and-HTTPS#default-tls-certificate]
Please let me know if there's anything I have done wrong.
thank you

Sorry to hear that @odssh!
Which OS are you calling cURL from?
What is the content of your /srv/inputs?

I am running the whole program on the ubuntu server and running the curl command from the same host on which the container is running.
When I checked the container and went to /srv/inputs directory inside the container it had none of the files that I mounted.
As you can see from the command I used I mounted the whole /inputs directory which contained three files:
cacert.jks  cacert.pem  conf.yaml
Please let me know if there's any other details you need
thank you

That's surprising. Can you try mounting that volume to another docker image and seeing if this time the directory contains the files?

I have tried to mount the inputs directory multiple times in all the ways possible but still the same error and works only when I give -k tag at the end. Should I try manually importing the files inside the container and then restarting the container?
thank you

So when you mount that directory to another container that uses a different image (not IBeam) you also don't see any files inside of the mounted directory? This would lead me to the following guesses:

there's something broken with the setup (eg. Docker, host)
the contents of /inputs are empty
the command has a typo or is invalid: eg -v /inputs:/svr/inputs

The latter two are easier to verify, but if these aren't the cause of the issue then I'm sorry but I'm not entirely sure how to help you. You could try the setup on a different machine with a fresh installation of Docker and see if the issue exists there too?

root@henon:~/gateway/inputs# curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem
curl: (60) SSL: unable to obtain common name from peer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
That is the error message I am getting now even after copying the ca certificate manually to the destination inside the container.
It might be because of the certificate is invalid or something else.
Also, how can I use the default certificate that comes with a gateway with python instead of that custom one. I went with the custom as the default certificate wasn't working with the python
thank you

I don't know how IBeam would behave if you copy the certificates manually - this is an unexpected behaviour and isn't really supported. How about the points I brought up in my previous reply?
The default certificate needs to be provided through the Inputs Directory, but since you can't mount the volume correctly this will not be possible until that gets sorted

root@henon:~/gateway/inputs# curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Hi, @Voyz
I got that error when using the beam's official docker image with the paper account. If possible can you please help me with where I am doing wrong?
thank you

Okay @odssh, I'm trying to help you but I'm feeling there's some miscommunication between us.  It's useful that you provide me with the error messages, but there's a number of questions or observations I replied you with that you haven't really addressed. Please let's do things in the right order and so we can figure out how to fix your setup, okay?
You need to start by discovering what is the reason for not being able to mount the Docker volume correctly. Have a look at this reply I made https://github.com/Voyz/ibeam/issues/8#issuecomment-813010891 and confirm for me which seems to be the case - or update me if this is no longer a problem and you can now mount volumes correctly. Otherwise please provide some screenshots of outputs of ls and pwd commands from your /inputs directory to verify its contents. Thanks 👍

Hi @Voyz thank you so much for your support. Finally got it to work with the custom SSL certificate.
Just another question I want to add, is how can I request the historical market data using that API? From the documentation, it seems like I need to make a get request to https://localhost:5000/v1/api/iserver/marketdata/history but when I do so I get the error message "Error 403 - Access Denied", Any idea on how can I fix that if you have done it before or if there's anything I am missing out?
thank you

Hey, happy to hear you got that sorted! Issues with Access Denied are described in the Troubleshooting:
https://github.com/Voyz/ibeam/wiki/Troubleshooting#access-denied
This should help you out 👍

Actually, it's working fine with the endpoint https://localhost:5000/v1/api/one/user but failing with the historical data endpoint. I think this issue is probably because of how I have written my request and have nothing to do with iBeam. Please let me know if there's any other endpoint you have worked with as I think I am providing the conid wrong. Any idea on how can I generate it?
thank you

You can check the conid's here:
https://contract.ibkr.info/v3.10/index.php?action=Stock Search&symbol=DDD&lang=en&ib_entity=&wlId=IB&showEntities=Y
Just type the symbol, click on 'Details' of the right stock and conid will be there.
Paste your full code for performing the request here, including the headers and parameters if you'd like me to see if I can spot something wrong with it.

Hi Folks,
I wanted to show some Node.js code for implementing the EXTERNAL_REQUEST without needing a external database/datastore:
Basically, you setup a Twilio account and get a phone number and then setup a SMS webhook to https://your-site.com/sms/incoming. Then paste in the following code to your Node.js express web server:
`
let smsToken;
router.get(/sms/code, async (req, res) => {
try {
if(req.query.token != 'ANY_STRING_VALUE_FOR_SECURITY') {
console.log('Invalid token specified in query param.');
res.send({});
return;
} else {
        console.log('request to /sms/code');

        smsToken = undefined;

        let waitCountMax = 120; // seconds
        let waitCount = 0;
        while(waitCount < waitCountMax) {
            waitCount++;
            if(!smsToken) {
                await new Promise(done => setTimeout(done, 1000));
            } else {
                break;
            }
        }

        if(smsToken) {
            console.log(`Responding with verificationCode: ${smsToken.verificationCode} obtained at ${smsToken.dateTime}`);
            res.send(smsToken.verificationCode);
            smsToken = undefined;
        } else {
            console.log('No SMS token received.');
            res.send({});
        }
    }
} catch(e) {
    console.error(e);
    res.send({});
}

});
router.post(/sms/incoming, async (req, res) => {
try {
// "Your requested authentication code: 289026"
let parts = req.body.Body.split(':');
    if(parts.length == 2) {
        let codePart = parts[1].trim();
        if(parts[0] == 'Your requested authentication code' && codePart.length == 6) {
            console.log(`Parsed Verification Code: ${codePart}`);

            smsToken = {
                verificationCode: codePart,
                dateTime: (new Date()).toISOString()
            };

        } else {
            console.log('Invalid verification code.');
            console.log(JSON.stringify(req.body));
        }
    } else {
        console.log('Non-verification code SMS detected.');
        console.log(JSON.stringify(req.body));
    }
} catch(e) {
    console.error(e);
} finally {
    res.writeHead(200, { 'Content-Type': 'text/xml' });
    res.end('<Response />');
}

});`
Next, start your docker container using something like:
docker run --env IBEAM_ACCOUNT=YOUR_PAPER_OR_LIVE_ACCOUNT_ID --env IBEAM_PASSWORD=YOUR_PWD --env IBEAM_TWO_FA_HANDLER=EXTERNAL_REQUEST --env IBEAM_EXTERNAL_REQUEST_URL=https://your-site.com/sms/code --env IBEAM_EXTERNAL_REQUEST_PARAMS={"token":"ANY_STRING_VALUE_FOR_SECURITY"} --env IBEAM_LOG_LEVEL=DEBUG --env IBEAM_PAGE_LOAD_TIMEOUT=500 -p 5000:5000 voyz/ibeam
I hope this helps someone... and also, great work Voyz !!!, you have saved a lot of people a lot of time =)

That's very useful @kashyappatel7 many thanks for sharing this! I'll add a link to this to the 2FA Wiki. And thanks for the kind words, I'm glad to help you find IBeam useful 😊

Hi, I tried to use Twilio to receive 2FA codes from IBKR but the 2FA messages don't even show up on Twilio REST API or their website. Apparently this is because virtual numbers cannot send SMS to other virtual numbers (IBKR sends 2FA from a virtual number). Is the Twilio solution still working for anyone?

@psyfb2 Sorry, I never used Twilio, couldn't comment on that. Google Messages solution works.


Failed to create application customizer 'ClientSideExtension.ApplicationCustomizer error that occurs randomly.

Target SharePoint environment
SharePoint Online
What SharePoint development model, framework, SDK or API is this about?
💥 SharePoint Framework
Developer environment
Windows
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[X] 💥 Microsoft Edge
[ ] 💥 Google Chrome
[X] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[ ] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details

browser version: 117.0.2045.41 (Official build) (64-bit)
SPFx version: 1.14.0
Node.js version: v14.17.0

Describe the bug / error
The extension idea is to render components in the SharePoint Online Search Page. One of the components, is a custom breadcrumb component that has a dropdown menu to switch between the SharePoint intranet hub sites. By switching between hubs, we change the search URL to search for the same keyword in the selected hub site.
It works fine however randomly it raises error in the browser console:
Error: Failed to create application customizer 'ClientSideExtension.ApplicationCustomizer. Error information is 'w.render is not a function'.
The error is occurring randomly when using the new rendered component (switching between hubs to search different hubs) by the SPFx extension.
Note: Sometime reloading the page helps to get the estension working again and rendering the components properly.
Errors screenshots:
Uncaught (in promise) Error: Failed to create application customizer 'ClientSideExtension.ApplicationCustomizer.12345678-1234-1234-1234-12345678'. Error information is 'w.render is not a function'.
customized-search-application-customizer_e80b452eb94107361745.js:2  An error occurred in addSearchPageMainHeader: TypeError: hn.render is not a function.

Uncaught (in promise) TypeError: hn.render is not a function.

An error occurred in addSearchResultCount: TypeError: hn.render is not a function.

Uncaught (in promise) TypeError: hn.render is not a function.

Steps to reproduce

The errors occur randomly after switching multiple times between hubs to search different hubs for the same keywork.
The steps to reproduce could be by rendering a component that would enable you to change the search URL to search different hub sites in the SharePoint online search page.

Expected behavior
The expected behavior is to have the extension working properly without getting random errors.

Is the error happening in your code or in PnP library code (or SPFx code -- doesn't appear to be SPFx)?

Hi Nick,
Yes, I'm still having the same issue and I don't have any ideas what could be done to fix the issue.
The source code should not be shared as you know it's my organization source code.
Regarding minimal repo, it not easy as the issue has been raised after my big change to implement the latest requirements and before my changes, I didn't get such issue.

Hi Nick,
You can close this issue as it turned out that another SPFx extension is causing this issue.
Thanks for helping.


fix link supported images in DEVELOPING doc

I'm closing this since I fixed this (and added more) to this file, and I also added additional docs also (all of this is due to the addition of OSSMC support): https://github.com/kiali/kiali-operator/pull/700


Executable location

Where is the app located? I'm trying to associate .xml files to this app and I can't find it anywhere in the store or on my hard drive. I have a shortcut in my Start Menu, but can't figure out where it's launching from.
Thanks.

Sorry, nevermind. Just needed to go to the Start Menu folder and select it.

Yeah, clickonce apps run from a sandbox located like this:
C:\Users\Chris\AppData\Local\Apps\2.0\60MPBT3D.ELV\H2QGHQ42.AJQ\xmln..tion_ab3ea86545595e2b_0002.0007_3f6d9d58fdf3015b\XmlNotepad.exe

You can find this using Task Manager right click on running XML notepad process and select Properties.  What we should do is add a menu item to XML notepad that says "Make Default App" or something that let's you pick the file extensions you want, and then XML notepad can do the magic for you... (since windows makes this so difficult, and gets it wrong).


update core rails and related libraries

updates rails to the latest dot release - it's a start #109

Thanks, will get this merged.


[FEATURE]: [NMI] Use connector_response_reference_id as reference to merchant

:memo: Feature Description

Reference id are used to map transactions in the connector’s dashboard.
Hyperswitch manages several reference ids, such as payment_id, attempt_id, and connector_transaction_id for a single transaction.
However, merchants may encounter uncertainty when determining which ID to utilize in the connector dashboard to identify the payment.

:hammer: Possible Implementation

When we receive a response from the connector for the payment, we deserialize it and populate the response field in RouterData.
For the TransactionResponse type, we must fill the connector_response_reference_id with a corresponding reference id for the merchant to identify the transaction.
One might need to have exposure to api docs of the connector for which it is being implemented to decide what to fill in connector_response_reference_id.
You can check this PR for further reference https://github.com/juspay/hyperswitch/pull/1735

🔖 Note: All the changes needed should be contained within hyperswitch/crates/router/src/connector/
:package: Have you spent some time checking if this feature request has been raised before?

[X] I checked and didn't find a similar issue

:package: Have you read the Contributing Guidelines?

[X] I have read the Contributing Guidelines

:sparkles: Are you willing to submit a PR?

😊 "I'd be delighted to work on this issue! Could you please assign it to me?"

Hey @TechWizard9999  , I see that you're already assigned to #2352. For now, we'll leave this issue open to any other interested contributors. You may pick this up once your work on #2352 is completed, and if no other contributors have expressed interest on this. I hope that's fine by you

Hey @AkshayaFoiger I want to work on this issue could you please assign this issue

Hey @parasss19, sure! I've assigned this to you.
In case you have any queries, you can ask them on this issue thread, or on our discord server, or on slack whichever you are comfortable with

Hey @AkshayaFoiger is there any docs for NMI?

Hello @parasss19 , I kindly suggest that you consult NMI's API documentation for reference and guidance.
In case you have any further queries, you can ask them on this issue thread, or on our discord server, or on slack whichever you are comfortable with

Hey @parasss19  ,
Let us know if you face any issues. Happy to help! : )

Hey @parasss19 , Let us know if you face any issues. Happy to help! : )

Yeah actually I am busy with some work I will send PR today

Sure @parasss19
Let us know if you face any issues.

Hey @parasss19  ,
Please let me know if you are working on it.
Thanks.

Hello @parasss19  , we didn't hear back from you, so we are unassigning you from this issue. If you are interested, we will assign you with other issues, if the current one is taken.

Hey @deepanshu-iiitu , can you assign this to me

@AkshayaFoiger @deepanshu-iiitu @VedantKhairnar @Sakilmostak
Could you please assign this issue to me? This will be my first contribution to this repo.

Hey @TejasMate , sure! I've assigned this to you.
In case you have any queries, you can ask them on this issue thread, or on our discord server, or on slack whichever you are comfortable with


Highlight 'lazy' as a keyword

lazy is a keyword, as found in the parser: https://github.com/facebook/reason/blob/985f28150026fb3ce353c699a2a0cbc9a52de919/src/reason-parser/reason_parser.mly#L4267

Thanks!


az sql server ad-admin create gives error

This is autogenerated. Please review and update as needed.
Describe the bug
az sql server ad-admin create -g {} -s {} -u {} -i {}`  - doesnt work , below error is displayed.
Command Name
az sql server ad-admin create
Errors:
create_or_update() missing 1 required positional argument: 'parameters'
Traceback (most recent call last):
python3.6/site-packages/knack/cli.py, ln 206, in invoke
    cmd_result = self.invocation.execute(args)
cli/core/commands/__init__.py, ln 608, in execute
    raise ex
cli/core/commands/__init__.py, ln 666, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
cli/core/commands/__init__.py, ln 659, in _run_job
    six.reraise(*sys.exc_info())
...
cli/core/commands/__init__.py, ln 306, in __call__
    return self.handler(*args, **kwargs)
azure/cli/core/__init__.py, ln 493, in default_command_handler
    return op(**command_args)
cli/command_modules/sql/custom.py, ln 2074, in server_ad_admin_set
    properties=kwargs)
TypeError: create_or_update() missing 1 required positional argument: 'parameters'

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az sql server ad-admin create -g {} -s {} -u {} -i {}

Expected Behavior
Environment Summary
Linux-4.15.0-1071-azure-x86_64-with-debian-stretch-sid
Python 3.6.5

azure-cli 2.2.0

Extensions:
storage-preview 0.2.8

Additional Context

sql

The issue seems to be caused bu wrong argument name in /azure/cli/command_modules/sql/custom.py
line 2074 should be:
parameters=kwargs instead of the current properties=kwargs

Fixed by #12541 . Thank you!

I'm facing the same issue. Is this in production?

No, it's not released yet :( next release is https://github.com/Azure/azure-cli/milestone/84


Alarm feed action should validate early if cron string is valid or not

From https://github.com/apache/incubator-openwhisk-package-alarms/blob/master/action/alarm.js#L25
The msg.cron is just taken and input is not validated
The cron npm package provides a way to check if pattern is valid, without starting the cron job
https://github.com/kelektiv/node-cron#how-to-check-if-a-cron-pattern-is-valid

this is complete

Sweet !


Tana paste Windows OS Terminal / Command line

Hello, @houshuang
Thanks for the Loom explanation!
Could you provide an example running this from a Windows Command line instead of Mac OS?
Regards
David

I don't have access to one, but I hope a community member using Windows can chime in?

I'll ask in the Slack Community. Thanks Stian!


Add state wise filter in the station page

✨ Feature Request
Description of the Feature
Add a filter on the station page that allows users to filter stations based on the state they are located in. This will help users find stations in a specific region more easily.
Problem Statement
Currently, users may find it difficult to search for stations based on their geographic location. Adding a state-wise filter will make it easier to locate stations in the desired state, improving user experience and navigation.
Proposed Solution
Implement a drop-down or multi-select filter on the station page that allows users to choose one or multiple states. Based on the selection, the list of stations will dynamically update to show only those that are located in the selected state(s).

[X] Yes
[ ] No

Go ahead @PraveenUppar


Deeply nested partials

When we include a jbuilder partial that is > 1 directory deep, e.g. in _order.json.jbuilder we want to include line items using:
json.line_items order.line_items, partial: "api/v2/orders/line_items/line_item", as: :line_item

However, jbuilder-schema ends up adding a $ref of "$ref": "#/components/schemas/line_item  rather than "$ref": "#/components/schemas/OrdersLineItemAttributes
That sometimes goes deeper, such as in the orders/invoices/_invoice.json.jbuilder, we have
json.line_items invoice.line_items, partial: "api/v2/orders/invoices/line_items/line_item", as: :line_item

which, again, just ends up with "$ref": "#/components/schemas/line_item rather than OrdersInvoicesLineItemAttributes

There were different options I was developing:

Guess the name from the :partial path, but there is a problem when a path includes api/v2 or something similar, I couldn’t find a way to get clear names in different situations yet.
Passing ref name in additional schema  parameter, but that also will take much more additional work not only to jbuilder-schema but to any dependent projects as well.
In the end I discovered that in some cases there is quite easy and elegant solution which already works: if we pass the name we want to :as option, for example as: :orders_invoices_line_item will produce OrdersInvoicesLineItemAttributes exactly as needed. Of course it will take some work to update variable names in the partials, but maybe it’s even better and they actually should be named like this?

PR #53 adds ref naming from :as option for all the cases, and when there is no :as option, it looks up for any locals passed and takes the first one:
json.line_items invoice.line_items do |line_item|
  json.partial! "api/v2/orders/invoices/line_items/line_item", orders_invoices_line_item: line_item
end


Unable to write URL entry with add command

Oops, I had erased the sentence I wrote by mistake.  😱 I'll fix it soon.

2955bc0


include: net: Add MSG_WAITALL flag definition

Add MSG_WAITALL socket flag required to support the corresponding NRF
counterpart flag, for translation within the offloading interface.
Linked to https://github.com/nrfconnect/sdk-nrf/pull/3116

@MirkoCovizzi please rebase

@rlubos please take a look


[WIP] markdown: Add support for new syntax ![title](url).

Can you add tests?  That's the most important part of any markdown change.

Can you add tests? That's the most important part of any markdown change.

@timabbott yes I am working on tests.

Hi @timabbott @andersk, this PR is ready for a review.


Fixes for interpolating vector valued functions

Codecov Report

Merging #13 (777f49c) into main (4160329) will increase coverage by 0.95%.
The diff coverage is 100.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #13      +/-   ##
==========================================
+ Coverage   96.83%   97.78%   +0.95%
==========================================
  Files           4        4
  Lines         631      633       +2
==========================================
+ Hits          611      619       +8
+ Misses         20       14       -6

Files
Coverage Δ

interpax/_spline.py
98.08% <100.00%> (+1.05%)
:arrow_up: