rp2040-sx1278

Can you make it support rp2040 development board?

This LoRa library built for Arduino framework. If you using Arduino, this library will worked. You should check that Arduino has RP2040 development board.
https://docs.arduino.cc/hardware/nano-rp2040-connect
Please note that this library has dependency with some Arduino core library including SPI library. So, clearly this library will not working outside Arduino development environment.

Update, I have check Arduino RP2040 core library here.
I think using this core library, LoRa library should work. But onReceive() and onTransmit() methods will not work because attachInterrupt() is not implemented in the RP2040 SPI library. It is described here.
Currently I don't have Raspberry pi pico or any RP2040 board. So, I can't test it yet.

thank you!  https://github.com/earlephilhower/arduino-pico  Arduino can support rp2040, but the current Lora libraries do not support rp2040 (raspberry PI Pico)，I designed a Lora + 2040 + LCD development board

Can you point out at which point this LoRa library does not support rp2040?
Some error message while you trying upload Arduino sketch, may be? So I can help to fix compatibility issue with rp2040.

Arduino:1.8.16 (Windows 10), ："Raspberry Pi Pico, 2MB (no FS), 125 MHz, Small (-Os) (standard), Disabled, Disabled, None, Pico SDK"
C:\Users\86186\AppData\Local\Arduino15\packages\rp2040\hardware\rp2040\1.12.0\libraries\LoRaRF\src\SX126x_API.cpp: In static member function 'static bool SX126x_API::reset(int8_t)':
C:\Users\86186\AppData\Local\Arduino15\packages\rp2040\hardware\rp2040\1.12.0\libraries\LoRaRF\src\SX126x_API.cpp:29:1: error: no return statement in function returning non-void [-Werror=return-type]
29 | }
  | ^

cc1plus.exe: some warnings being treated as errors
exit status 1
thank you! I choose Pico to compile, but I can't compile

Try to modify function type from bool to void in SX126x_API.cpp and SX126x_API.h files located at LoRaRF library folder.
Change in SX126x_API.cpp
bool SX126x_API::reset(int8_t reset)
{
    pinMode(reset, OUTPUT);
    digitalWrite(reset, LOW);
    delay(1);
    digitalWrite(reset, HIGH);
}

to
void SX126x_API::reset(int8_t reset)
{
    pinMode(reset, OUTPUT);
    digitalWrite(reset, LOW);
    delay(1);
    digitalWrite(reset, HIGH);
}

and in SX126x_API.h
static bool reset(int8_t reset);

to
static void reset(int8_t reset);

thank you! I just used https://github.com/jgromes/RadioLib This library can accept data, but it must be connected to sx1278 dio1 pin


Issue with OpenAI tokenizer not functioning as expected in streaming mode

System Info
I am developing an app utilizing the streaming functionality in Python 3.10. I've crafted my own generator and callback for this purpose. However, I've stumbled upon an issue when setting llm.streaming to true and utilizing a callback - the OpenAI tokenizer appears to malfunction, displaying a token count of 0$ which is not the expected behavior.
Who can help?
@agola11
@hwchase17
Information

[ ] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[X] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[X] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[X] Callbacks/Tracing
[ ] Async

Reproduction
def arun(self, g, chat_history, query:str, config:dict, agent_id:str):
    try:
        llm = ChatOpenAI(
            temperature=0,
            verbose=False,
            model='gpt-4',
            streaming=True,
            callbacks=[ChainStreamHandler(g)]
        )
        with get_openai_callback() as cb:
            custom_agent = customAgent.run(g, llm, chat_history, query, config, agent_id)
            print(cb)
            return custom_agent
    finally:
        g.close()

Tokens Used: 0
Prompt Tokens: 0
Completion Tokens: 0
Successful Requests: 0
Total Cost (USD): $0.0
Expected behavior
I aim to have the tokenizer working correctly to gauge the token usage and its corresponding cost. Any guidance or suggestions to rectify this problem would be greatly appreciated

I'm getting same problem...
chain = LLMChain(
    llm = ChatOpenAI(
        streaming=True
    ),
    prompt = ChatPromptTemplate.from_messages([('human', "What should I call my store that sells {product}")])
)
with get_openai_callback() as cb:
    output = chain.run(product="paper airplanes")
print(cb)

yields:
Tokens Used: 0
Prompt Tokens: 0
Completion Tokens: 0
Successful Requests: 0
Total Cost (USD): $0.0
But:
chain = LLMChain(
    llm = ChatOpenAI(
        streaming=False
    ),
    prompt = ChatPromptTemplate.from_messages([('human', "What should I call my store that sells {product}")])
)
with get_openai_callback() as cb:
    output = chain.run(product="paper airplanes")
print(cb)

yields
Tokens Used: 90
Prompt Tokens: 17
Completion Tokens: 73
Successful Requests: 1
Total Cost (USD): $0.0001715

@OrelOfficely @rsarvis were you able to figure out how to get token tracking working when you have streaming = True?


DDP+ manual optimization support

Gradients aren't being synchronised if using manual_backward

I was just wondering whether the issue I've just added is duplicated with this one? Or that is another bug? Thanks!


use removeAllEntities instead of reset to clear database when it is t…

…oo large

As mentioned in the related issue, this isn't perfect as tombstoned data is left over. However, without something like this, stores will treat already-loaded data as the source of truth which will prevent the reset() data from actually being removed. LGTM!

I'll merge this, then try do an import and verify manually. Jason, let me know what you think.


Request hangs if file not sent

Hi, is there any way to get skipper-s3 to just ignore the S3 upload if a file by the name of req.upload(name) is not present in the request? Right now the request just hangs open, I'd like it to just silently ignore the upload if no file was sent. I'm looking around online but not finding many options on how to check for the file ahead of time.

Howdy @mattcodez,
Since this is a question rather than a bug report, StackOverflow would be a better platform.
Thanks!
Best, Irl


[documentation] Add to docs limitations of CDSDashboards for 0.4.0 release

Only the environments specified in qhub-config.yaml will show up in cdsdashboards. This issue will be properly address in 0.4.x.

Hey @costrouc I don't recall what these limitations were? Perhaps we can include them in the next release?

The limitation is that the environments are not dynamic and the only ones that show up are the ones in the qhub-config.yaml.


OpenGL texture binding to HTML canvas

Hi,
Awesome project, worked out of the box! (only issue I have is that "Home" routing does not work)
I'm new to WebAssemblies (and to be honest, to Angular as well) but have a fairly good understanding of how things work and bind together (for both frameworks).
I'm trying to understand how the 3d-cube example works, but I can't figure out a few things:

How is main(argc, argv) being invoked?
After writing the above question I paused writing and started digging a bit more to figure this one on my own, so it would be nice to get a confirmation:

Each of the objects extend EmWasmComponent which calls instantiateJs to creates a script element in the HTML document which reference the path to the JS file that was created by emcc. This JS code invokes main

How is the canvas of the 3d-cube bound to OpenGL's texture?

Thanks

Hi @zivsha

To initialize a WebAssembly module, you need to call WebAssembly.instantiate. When compiling C/C++ code with emscripten, you get generated JavaScript glue code to initialize your WebAssembly module instead. This generated JavaScript glue code calls main (if exists).

As WebAssembly does not support direct access to Web APIs or DOM, you need to wrap them in JavaScript calls. Emscripten generates such wrapped glue code, so that it is transparent for the C/C++ developer. In C/C++ you would use OpenGL, which gets "translated" to WebGL calls. You can see these calls in the generated JavaScript glue code too.

@boyanio thank you!
So in your C++ code you create a window with a size of 500x500 which renders the cube and texture using (eventually) WebGL, and emscripten creates the glue to the DOM but does it know where to position the window? what am I missing here?

There are different OpenGL versions. There is no 1:1 mapping between OpenGL and WebGL, you can read about Emscripten's support for OpenGL here.
The code for the 3D cube is not written by me, I have found it somewhere on the web and I have modified it a bit to compile it to WebAssembly. On the web it makes no sense to manipulate windows, so what happens in reality is you manipulate the canvas. You can see the source code for glutInitWindowSize here.

It is possible to have multiple OpenGL applications compiled to WebAssembly and each of them will use a separate canvas

Thank you!


fix(mqtt): fix eslint and support mqtt subscribe function

:tada: This PR is included in version 1.57.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fixed navigateToAddPaypalRoute issue

Details

Fixed Issues

$ https://github.com/Expensify/App/issues/16287
PROPOSAL:
Tests

Click on 'Profile' avatar
Click on 'Payments' link
Click on 'Add payment method' button
Click on 'Paypal.me' button
Enter user name & click on 'Add PayPal account' button
Click on the newly added Paypal account

[x] Verify that no errors appear in the JS console

Offline tests

Click on 'Profile' avatar
Click on 'Payments' link
Disabled 'Add payment method' button

QA Steps

Click on 'Profile' avatar
Click on 'Payments' link
Click on 'Add payment method' button
Click on 'Paypal.me' button
Enter user name & click on 'Add PayPal account' button
Click on the newly added Paypal account

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

Mobile Web - Chrome

Mobile Web - Safari

Desktop

iOS

Android

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos
Straightforward one as it's a typo added web alone to confirm it's not crashing

Web
https://github.com/Expensify/App/assets/85645967/042763e3-bd17-4c52-90c3-1f1d6e141d90

@multijump Include this as Test Steps:
Click on 'Profile' avatar
2. Click on 'Payments' link
3. Click on 'Add payment method' button
4. Click on 'Paypal.me' button
5. Enter user name & click on 'Add PayPal account' button
6. Click on the newly added Paypal account

@Li357 Thanks, yeah sorry about that!
@multijump Can you please update the steps soon, so this doesn't block QA from testing, thanks!

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

@Li357 Thanks, yeah sorry about that!
@multijump Can you please update the steps soon, so this doesn't block QA from testing, thanks!

Hi, @Santhosh-Sellavel
Updated the QA steps.
Thanks

🚀 Deployed to staging by https://github.com/Li357 in version: 1.3.48-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/puneetlath in version: 1.3.48-5 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to staging by https://github.com/Li357 in version: 1.3.49-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/Beamanator in version: 1.3.49-3 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


improved dvid backend support

DVID does not have a specific datatype for multi-scale chunk data.  However, a power of two pyramid is created by convention.  For example, for a data instance 'grayscale', 'grayscale_1', 'grayscale_2', etc represents multiple levels.  I made a small modification to the dvid frontend to check for this convention.
As noted by the author, DVID also does not indicate which tiles are available for its imagetile datatype (XY, YZ, and/or XZ).  However, by default, DVID returns a blank tile if nothing is available.  I changed the default to expect all orientations given that the performance is really slow anyway for XZ and YZ if only XY is assumed.
I added some basic parsing of the DVID dag so that the version history can be observed (segmentation can now be viewed at different points in time).  In the future, it might make sense to allow the user to specify the repo id and then choose a specific version.  For now, I just show all the version nodes for every repository in DVID.

I signed the CLA.
From: googlebot<EMAIL_ADDRESS>Sent: Monday, June 20, 2016 2:26 PM
To: google/neuroglancer
Cc: Plaza, Stephen; Author
Subject: Re: [google/neuroglancer] improved dvid backend support (#4)
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
📝 Please visit
https://cla.developers.google.com/ to sign.
Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.
If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check
your existing CLA data and verify that your
email is set on your git commits. If you signed the CLA as a corporation, please let us know the company's name.
—
You are receiving this because you authored the thread.
Reply to this email directly,
view it on GitHub, or
mute the thread.

Thanks for these improvements, Stephen!
I just have a few small comments.
For style consistency, can you please run clang-format on all changed files and ensure that tslint gives no errors?  You can npm install both clang-format and tslint.  (The npm version of clang-format is kept up to date.)  In particular, I'd like to maintain the use of single quotes for string literals === and !== instead of == and !=, except in cases where the behavior of == and != is specifically needed.  Also the interface definitions don't need a trailing semicolon (actually an existing mistake I made in base.ts), which becomes apparent once you run clang-format.

Thanks Jeremy for the feedback!  I believe I have addressed all of your comments and ran clang-format followed cslint for the three changed files.  (I originally thought that clang-format was going to remove the semicolon from the interface hence the extra commit.)

Great, thanks!


Add filePath property to persist cropped image to disk

The captureFrame method of Surface in gl-react-native has the ability to specify file as a format and filePath to persist the cropped image to disk/storage.
Update line 196 in ImageCrop.js to:
return this.refs.cropit.captureFrame({quality: this.props.quality, type: this.props.type, format: this.props.format, filePath: this.props.filePath})
Example usage:
let filepath = ${RNFetchBlob.fs.dirs.CacheDir}/temp_post_image.jpg;
<ImageCrop
ref         = {'cropper'}
format      = {'file'}
filePath    = {filepath}
image       = {this.state.image}
cropHeight  = {Screen.width}
cropWidth   = {Screen.width}
type        = 'jpg'
quality     = {0.6}
maxZoom     = {80}
minZoom     = {20}
panToMove   = {true}
pinchToZoom = {true}
/>

feature added
https://github.com/stoffern/react-native-image-cropper/blob/master/src/ImageCrop.js#L195


Use new loading buttons styles

Uses the new loading buttons in Builder as requested in #12132.
Example:

Places where it has to be updated:

[x] Run analysis.
[x] Update the map.
[x] Publish
[x] Apply CSS.
[x] Export as an image.
[x] Apply SQL.

How to test
Check the list above. In addition to this, check if we should update the analysis quota view as well.

Only 17 files? 😱 :joy:

good catch with that button @nobuti. This button should be on every place that we disable the CTA while loading. cc/ @urbanphes

@nobuti @saleiva so after spending ~5h installing the stuff to get the analysis quota view button working, turns out that that button disappears when the analysis starts...so I guess no need for a state change in that button, right?
But hey we got the dataservices-api working, which rocks 🤘

Sorry guys, I was not sure, that's why I suggested to check it in acceptance 😞

but while it's being check, the button displays a loading indicator, right?

While the number of credits that it will cost is being checked, there's no button, just a panel with a loading indicator. Same thing when you hit 'confirm', the button is replace with a 'Running Analysis' loading indicator in a panel.

Agree, in order to apply the button, we need a revamp of that view.

Testing in ded02.

It works pretty well 🚀


[IDSEQ-2509] Searching in DD, if the term is deleted there's no visual cue that it's still being applied

Description
Searching in DD, if the term is deleted there's no visual cue that it's still being applied
The intended change, motivation and consequences.
Added to state* (not prop, sorry got it wrong in commit msg) lastSearchedTerm and onBlur method to fix bug.
Notes
Optional observations, comments or explanations.
Tests

Server-side code should have automated tests
Client-side code and scripts should have at least a manual test plan
See also IDseq PR Guidelines

Looks good to me! Thanks @ovalenzuela19!

@ovalenzuela19 Please write down the tests, as they also serve as instructions for the QA team. Or refer to the IDseq ticket if that has enough information.


Context type discussions

Upon diving into the source code, and finding a situation where this would be imperative to improve the runtime of my program, I ran into 2 potential issues, or lack of knowledge of any workarounds.
I saw that Context types of candidate_subclass args are always set to Span in the database model. From my understanding, this can be any Context type (Sentence, Paragraph, etc.) but hasn't been implemented to do so. Is there another way to define this without manually editing the database?
On the same token, CandidateExtractor.apply() strictly generates TemporarySpans to represent candidates, which would disregard any changes to the Context type of any candidate_subclass arg. I also wonder if it's current possible to specify the Context that the CandidateExtractor considers to match the candidate model.
If I am wrong, please correct me, I am still attempting to understand the full workflow of this tool.
Edit: I realize that Ngrams.apply() is the ultimate cause of the TemporarySpan issue, but I also realize an implementation that I overlooked would accomplish what I am looking for. Also discovered that the context type changes throughout method calls, one requiring a Document type while the next requires Sentences. If it's in the works to customize the Context type of a candidate model, this may need to be looked at as well.

I saw that Context types of candidate_subclass args are always set to Span in the database model. From my understanding, this can be any Context type (Sentence, Paragraph, etc.) but hasn't been implemented to do so. Is there another way to define this without manually editing the database?

Right now, fonduer supports Span and Image as members of candidate_subclass, and the one you observed is because you define candidate_subclass with Span.

On the same token, CandidateExtractor.apply() strictly generates TemporarySpans to represent candidates, which would disregard any changes to the Context type of any candidate_subclass arg. I also wonder if it's current possible to specify the Context that the CandidateExtractor considers to match the candidate model.

This is user-defined. If users want to have different types of Contexts to extract, they need to update the MentionSpace to support that.

I have 2 questions then. Would it be feasible to expand the scope of possible contexts for candidate_subclass? And also, seeing how there's not much documentation for this project, where can I define a Context to extract? I have not come across MentionSpace in any of my exploring through the source code.
Note: for most cases where users are asking for clarity such as this, until there is a full set of documentation for how these things are possible, it might be necessary to give more insight on how these things are accomplished. I personally have very little Python experience, so exploring and understanding the source code can be exhausting and takes a lot of time out of the task that I'm using Fonduer for, but it's also my only option to build good foundational knowledge on its functionalities.

I have not come across MentionSpace in any of my exploring through the source code.

It is going to be renamed MentionSpace in v0.3.0. In v0.2.3 it is called CandidateSpace.
To see some more of the functionalities, including defining a custom MentionSpace/CandidateSpace, i recommend checking out the Tutorials. Althought we do not have any tutorials in expanding the scope of possible contexts to extract, we do show custom MentionSpace/CandidateSpaces for the hardware application.

Just to share you more information.

Would it be feasible to expand the scope of possible contexts for candidate_subclass?

The answer is YES. The users can customize the CandidateSpace to the information they want to extract. As Luke mentioned, we have tutorials to show how to use it and there is a hardware_imamge tutorial to show how to extract images as candidates. In the v0.3.0, here are the pointer to how we build these functionalities.

And also, seeing how there's not much documentation for this project, where can I define a Context to extract?

We are pushing to have more documentation in the next release (v0.3.0).
Thanks for your comments/feedback!!!

Thanks! I understand how the CandidateSpace the user wants to extract is customizable, but only to the extent of Spans and Images. Do you foresee this expanding further, i.e. Sentences?

Some of our collaborators' applications extract candidates based on the full sentence. So I would say, Yes. And it should be easy to extend to other objects well.

I am having a similar issue. In my case the spans can not just be in the cells of a table but also in the URL of an image or the document name. I wrote a quick and dirty custom Mentionspace and MentionRegex:
from fonduer.candidates.models.temporary_context import TemporaryContext
from fonduer.candidates import MentionNgrams, MentionDocuments, MentionFigures
from typing import Collection, Optional, Iterator

class StationMentionSpace(MentionNgrams):
    """Defines the **space** of Mentions.

    Defines the space of Mentions as all n-grams (n_min <= n <= n_max) in a
    Document *x*, divided into Sentences inside of html elements (such as table
    cells).

    :param n_min: Lower limit for the generated n_grams.
    :type n_min: int
    :param n_max: Upper limit for the generated n_grams.
    :type n_max: int
    :param split_tokens: Tokens, on which unigrams are split into two separate
        unigrams.
    :type split_tokens: tuple, list of str.
    :param types: If specified, only yield TemporaryFigureMentions whose url ends in
        one of the specified types. Example: types=["png", "jpg", "jpeg"].
    :type types: list, tuple of str
    """

    def __init__(
        self, n_min: int = 1, n_max: int = 5, split_tokens: Collection[str] = [], types: Optional[str] = None
    ) -> None:
        """Initialize MentionNgrams."""
        MentionNgrams.__init__(self, n_min=n_min, n_max=n_max, split_tokens=split_tokens)
        if types is not None:
            self.types = [t.strip().lower() for t in types]
        else:
            self.types = None

    def apply(self, doc: Document) -> Iterator[TemporaryContext]:
        """Generate MentionNgrams from a Document by parsing all of its Sentences.

        :param doc: The ``Document`` to parse.
        :type doc: ``Document``
        :raises TypeError: If the input doc is not of type ``Document``.
        """
        if not isinstance(doc, Document):
            raise TypeError(
                "Input Contexts to MentionNgrams.apply() must be of type Document"
            )

        for ts in MentionNgrams.apply(self, doc):
            yield ts

        for ts in MentionDocuments.apply(self, doc):
            yield ts

        for ts in MentionFigures.apply(self, doc):
            yield ts

from fonduer.candidates.models.document_mention import TemporaryDocumentMention
from fonduer.candidates.models.figure_mention import TemporaryFigureMention
from fonduer.candidates.models.span_mention import TemporarySpanMention
from fonduer.candidates.matchers import RegexMatchSpan

class RegexMatchFull(RegexMatchSpan):
    """Matches regex pattern on **full concatenated span, document-name and figure-urls**.

    :param rgx: The RegEx pattern to use.
    :type rgx: str
    :param ignore_case: Whether or not to ignore case in the RegEx. Default
        True.
    :type ignore_case: bool
    :param search: If True, *search* the regex pattern through the concatenated span.
        If False, try to *match* the regex patten only at its beginning. Default False.
    :type search: bool
    :param full_match: If True, wrap the provided rgx with ``(<rgx>)$``.
        Default True.
    :type full_match: bool
    :param longest_match_only: If True, only return the longest match. Default True.
        Will be overridden by the parent matcher like :class:`Union` when it is wrapped
        by :class:`Union`, :class:`Intersect`, or :class:`Inverse`.
    :type longest_match_only: bool
    """

    def _f(self, m: TemporaryContext) -> bool:
        """The internal (non-composed) version of filter function f"""

        def apply_rgx(attrib_span):
            # search for string as e.g. "_" split operator is used
            return (
                True
                if self.r.search(attrib_span)
                is not None
                else False
            )
        if isinstance(m, TemporarySpanMention):
            return RegexMatchSpan._f(self, m)
        if isinstance(m, TemporaryFigureMention):
            return apply_rgx(m.figure.url)
        if isinstance(m, TemporaryDocumentMention):
            return apply_rgx(m.document.name)

        raise ValueError(
            f"""
            {self.__class__.__name__} only supports
            TemporarySpanMention, TemporaryFigureMention and TemporaryDocumentMention
            """
        )

It does extract the DocumentMentions and FigureMentions next to the SpanMentions.
However, I ran into two problems:

It will mention the whole document/figure and I can not refer only a span of the name/URL. Thus I would have to apply parser functions each time I use the candidate later e.g. to compare against the gold set or extract the final data.

Even though the mentions are extracted, somehow the candidate extractor does not build any candidates from the DocumentMentions and FigureMentions. My candidate extraction code is similar to the hardware-tutorial.

I will look more into it, but I would like to know if there is a simpler way to reference these spans? And what might fail the candidate extraction?


recommended preprocessing for video model

for the pretrained webcam/custom video model, is it recommended to always resize to 672x512 ? or can i use the same preprocessing as for the photographic model (https://github.com/ZHKKKe/MODNet/blob/master/demo/image_matting/colab/inference.py)? for example, if my video is 16/9 instead of 4/3, or god forbid, vertical video, would it be better to use the latter preprocessing that would more closely preserve the aspect ratio?

Hi, thanks for your attention.
Q1: for the pretrained webcam/custom video model, is it recommended to always resize to 672x512 ?
Nope. I think the original video ratio should be used. Sorry that We forgot this point before.
We will work on this and update the demo in these two days.

@eyaler The bug is fixed. Thanks,

@eyaler The bug is fixed. Thanks,

there is still some difference from https://github.com/ZHKKKe/MODNet/blob/master/demo/image_matting/colab/inference.py where the logic is applied only when if max(im_h, im_w) < ref_size or min(im_h, im_w) > ref_size. is that more appropriate for the photographic model but not for the video model?

there is still some difference from https://github.com/ZHKKKe/MODNet/blob/master/demo/image_matting/colab/inference.py where the logic is applied only when if max(im_h, im_w) < ref_size or min(im_h, im_w) > ref_size. is that more appropriate for the photographic model but not for the video model?

Nope. I found that the effect of the conditional statement if max(im_h, im_w) < ref_size or min(im_h, im_w) > ref_size on the result is negligible. Therefore, I did not add it in the video demo.

Nope. I found that the effect of the conditional statement if max(im_h, im_w) < ref_size or min(im_h, im_w) > ref_size on the result is negligible. Therefore, I did not add it in the video demo.


Task 3

Who is going to do task 3? We can discuss on how to proceed.

@tirthbha is doing it

@tirthbha have you started doing it ?
How are you planning to do the comparison?
In task 3, there is a task:
Compare the numpy integration functions to the ones found in compphys
So what really are we comparing? Can you elaborate?

@reshnashrestha Have you done the task1 and task2?

I overviewed the task. Group members are working on it. I have generated the library with all functions. Surgically I think that task 3 can be done to some extent without huge dependency in other tasks so if it’s possible we can proceed. Have you started?

@reshnashrestha I am trying to figure out the issue that I encountered for the integral calculation, which is as follows:
Although I handled the singularity at x=0 for the functions: exp(-1\x)[return np.where(x != 0, np.exp(-1/x), 0)] and cos(1/x) [return np.where(x !=0, np.cos(1/x),0)]
I am getting the warning
"RuntimeWarning: divide by zero encountered in divide
How have you handled this issue in your case?

@reshnashrestha We can find out the roots by using simplest method like Bisection  method and Newton-Raphson method, right?

You are right. Which methods do you adopted? Are they working? Do we need to make any changes?

@reshnashrestha Yes both are working efficiently.

That’s great.

What is your update on task 3

@reshnashrestha Are we supposed to compare the results obtained from the three methods ( in task1) with those obtained from numpy integration?

@reshnashrestha Have you done first point of task 3?

Do you need any help?

Everything is well now!


Replace φ special character

According to #52 the special character is problematic, so I’ve replaced it here.

probably would make more sense using phi or gr

Same issue:
https://github.com/jxnblk/grid-styled/pull/53

Thanks for this!

This would mean a breaking change
Most apps should probably be setting <meta charset='utf-8'> IMO

That said, using a unicode character is completely unnecessary, so I'll likely keep this PR open until preparing for the next major version.

Ended up just removing the Golden component altogether in v3, but thanks again for the PR!


Explanation for the tables

Hi, what does "Total spans in test set"  mean in the tables?
BTW, i got similar results on conll2003 data.
It seems bert is not so effective for NER.

Span is a concatination of the sequence: B_ORG, I_ORG, I_ORG => ORG. There are counting of this spans in the table. In all articles we can see metrics for IOB (as conll-2003, with B_ORG and so on). For example Google achived ~0.967 on conll-2003 (as we can see in their paper).
I think, that BERT is very effective and universal for all tasks :+1: .


Feat/24 add list functions

Pull Request: Implement New Listing Features with Tests
Description
This pull request includes the following changes:
Updates to README
Added new method descriptions:

list_files(): Lists remote tax rates with their attributes.
list_tax_rates(): Lists remote files, their attributes, and Unix-style paths.
list_accounts(): Lists remote accounts with their attributes.
list_journal_entries(): Lists remote journal entries with their attributes.

New Constants File

Created a separate file for constants that consist of CATEGORY_COLUMNS, FILE_COLUMNS, TAX_COLUMNS, ACCOUNT_COLUMNS, and JOURNAL_ENTRIES.

Added Methods

list_tax_rates(): Lists remote tax rates with their attributes.
list_accounts(): Lists remote accounts with their attributes.
list_journal_entries(): Lists remote journal entries with their attributes.

New Unit Tests
Added unit tests to check that listing methods return expected columns with correct data types:

Test for list_tax_rates():

Checks that the DataFrame returned by CashCtrlClient contains all expected columns with the correct data types as specified in TAX_COLUMNS.
Ensures there are no unexpected columns and verifies the data types for expected columns.

Test for list_accounts():

Verifies that CashCtrlClient's list_accounts() returns a DataFrame containing expected columns with the correct data types.
Ensures no unexpected columns are present and checks data types for expected columns.

Test for list_journal_entries():

Validates that CashCtrlClient's list_journal_entries() contains expected columns with correct data types as specified in JOURNAL_ENTRIES.
Ensures no unexpected columns are found and checks data types for each expected column.

ChatGPT Review
 Click for ChatGPT prompt
Overview
After reviewing the proposed changes, I analyzed how they align with community standards and best practices in open-source projects. This includes considering code structure, documentation, unit tests, and overall maintainability. Below are my observations and suggestions for improvement.
Code Structure

Current Practice: The code structure seems consistent, with logical separation of constants, methods, and tests.
Comparison: Compared to established open-source projects, this structure aligns well. Notable projects like [Project A] and [Project B] have similar modular approaches.
Improvement Suggestion: Ensure consistent naming conventions across all modules and methods. This helps maintain clarity and readability.

Documentation

Current Practice: The README has been updated to include new method descriptions, which is a good start. The addition of comments in the code also contributes to readability.
Comparison: Many successful open-source projects, such as [Project C], provide detailed examples or usage guides in their documentation.
Improvement Suggestion: Consider adding code snippets or examples to the README to illustrate how the new methods can be used. This helps new contributors understand the context better.

Unit Tests

Current Practice: The inclusion of unit tests to verify the expected columns and data types is a positive step toward ensuring code reliability.
Comparison: Leading open-source projects often include extensive test coverage, sometimes using Continuous Integration (CI) for automated testing.
Improvement Suggestion: Consider implementing a CI setup (like GitHub Actions or Travis CI) to automatically run tests upon code changes. This can help maintain code quality over time.

Community Interaction

Current Practice: The pull request contains clear descriptions and explanations of the changes, which is beneficial for community collaboration.
Comparison: Open-source projects with active communities often have clear guidelines for contributing, issue tracking, and pull request review.
Improvement Suggestion: Consider creating a CONTRIBUTING.md file to outline best practices for contributions, including coding standards and testing requirements. This can encourage more consistent and collaborative contributions.

Conclusion
Overall, the proposed changes align with common open-source best practices. The focus on documentation and unit tests is a positive step. Implementing a CI system and providing more detailed examples in the documentation could further enhance the project's quality and community engagement. Thank you for considering these suggestions. I look forward to the final implementation.

Notes for Reviewers
Please review the new methods and tests for accuracy and provide any feedback or suggestions for improvement.

Merged. Thanks a lot!


Grey screen in Steam, frame rate immeasurably slow

Hi ALVR! I just installed today's version of everything, updated my GPU drivers, and gave it a shot.
SteamVR detects the headset and the mirror window shows head movement and controller movement, but the image on the Oculus Go is just one big grey screen with compression artifacts. I logged into my router to watch traffic and it was transferring data at 3.8 MBPS (so almost 30 megabit on the nose) so it must be a decompression issue on the Oculus Go? Head movement within the VR environment inside the headset works in that it moves the compression artifacts pretty close to 1:1 movement in the grey screen.
Any thoughts?

Source system: 6700K @ 4.7GHz, 2x980 Ti (tried with SLI enabled and disabled.)
Render speed in the Steam preview window is plenty fast. Latest nvidia drivers as of today.

Do you get same issue whenever restart client?
Please try adjust video bitrate or buffer size. And try to press "Cause packet loss" button on Debug tab.

I have yes, but given another user's report of issues with dual GPUs even with SLI disabled, I am going to attempt to install the driver on another system to see if a single-card system will handle it better.

Okay further testing:
Totally separate PC on the same wired network<EMAIL_ADDRESS>1080 Ti) and same wifi. Same Oculus Go. Same result.  Tried about 10 different settings between 1mbit and 100mbit (restarting the server each time) and no appreciable change. It looks like it is simply having difficulty decoding the stream. Communication from headset to server seems fine: the movement of my head is reflected instantly in the VR mirror window.
Ran a couple internet speed tests in the Go's browser. Even the lowest of the results was above 30mbit so that should rule out a bandwidth issue. Router is an AC68U that's older, but still stable most of the time (though showing some drops and errors to my nvidia TV hield from time to time - maybe showing signs of age) and within 8 feet line-of-sight to the Oculus Go.
Logged into the router: When set to 30mbit, the stream of the grey SteamVR home (which is, I assume, the source of my artifact-filled not-moving grey screen) pushes about 21 mbit and that's reflected in the traffic monitor to the 5GHz link reporting 2,671 KB/s.
Checking CPU power on the Oculus - without rebooting played Dead and Buried and Tomb Raider VR games without a hitch, plus some some low-quality streaming from Within and some high-quality streaming locally on my network using SkyboxVR. I even stuck a heatsink to the front of the headset and plugged it into the wall to make sure it wasn't a power/heat issue.
I was thinking of replacing that router anyway, but am open to further suggestions.

while streaming does alvr server show any packet loss?

Good question:
I checked the Go's Wifi and it's reporting a 5GHz link, "excellent" quality, and 866mbit negotiated speed
OH YES does it show packet loss! Wow it spikes up HIGH. It's saying it's losing orders of magnitude more packets than it is sending:  https://i.imgur.com/ycKSXFU.png
__

hell... your network is messy. try changing qos settings. check main page for guide

https://www.reddit.com/r/theNvidiaShield/comments/2rkzgr/shield_portable_gamestream_tips_that_helped_out_a/
also raise the buffer to 1mb

This was fun to record:
https://youtu.be/uR3rIOz9LHM
QOS was disabled on my router. The only QOS options are for internet shaping and not for LAN/wireless traffic shaping.

Well can't help you there :P
You need to try different wifi channels / settings, move router etc I guess

I just tried iperf to look at the UDP on my wifi and it's just awful. No
router settings have helped thus far. I guess #closed
Connecting to host <IP_ADDRESS>, port 5201
[  4] local <IP_ADDRESS> port 57415 connected to <IP_ADDRESS> port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   152 KBytes  1.24 Mbits/sec  19
[  4]   1.00-2.00   sec   120 KBytes   985 Kbits/sec  15
[  4]   2.00-3.00   sec   136 KBytes  1.11 Mbits/sec  17
[  4]   3.00-4.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   4.00-5.00   sec   112 KBytes   918 Kbits/sec  14
[  4]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   6.00-7.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   7.00-8.00   sec   136 KBytes  1.12 Mbits/sec  17
[  4]   8.00-9.00   sec   144 KBytes  1.18 Mbits/sec  18
[  4]   9.00-10.00  sec   136 KBytes  1.12 Mbits/sec  17

[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  4]   0.00-10.00  sec  1.29 MBytes  1.08 Mbits/sec  0.146 ms  0/164 (0%)
[  4] Sent 164 datagrams
iperf Done.
On Sun, Jun 24, 2018 at 12:39 AM, Iosif Karaioannoglou <
<EMAIL_ADDRESS>wrote:

Well can't help you there :P
You need to try different wifi channels / settings, move router etc I guess
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/polygraphene/ALVR/issues/85#issuecomment-399736245,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFw0ALUBhjymbv6ZkrUKm5-0goxYWLb-ks5t_0ImgaJpZM4U0V3y
.

Thank you for executing various checks!
The broken "packet loss" counter is caused by out-of-order packet. ALVR client reports negative value as lost packet on bad ordered packet. And the GUI shows broken values.
I will fix the broken counter on future release. And I want to add packet re-ordering function to recover by this types of network error.
Current ALVR implementation is very weak for continuous out-of-order packet.

Chalking this one up to a router not properly handling UDP traffic.  Closing. Thanks for all the help!

Replaced flaky AC-68U with new AC-86U. Buttery smooth movement now. Thanks!


Detect CNAME loops in the CNAME cache while trying to resolve

Motivation:
We need to detect CNAME loops during lookup the DnsCnameCache as otherwise we may try to follow cnames forever.
Modifications:

Correctly detect CNAME loops in the cache
Add unit test

Result:
Fixes https://github.com/netty/netty/issues/10220

/cc @igor-lukyanov

@slandelle I think it should be good enough as A would be in the Set and so we would break. What I am missing ?

My bad, indeed, I had missed the check on the boolean returned by add.

Then, your call: perfect solution at the cost of extra HashSet allocation vs loop limit without extra allocation?

@slandelle lets keep the set for now...we can add extra options later on. We only need the set if we have more then 2 links in the chain anyway

Can also consider this loop detection algorithm: https://github.com/google/guava/blob/48c7cfd5e38c8492e4356aab5c633254d1ce43be/guava/src/com/google/common/base/Throwables.java#L255

@Bennett-Lynch thats a neat trick...


Invalid URL in manifest

[SIM] Validating plugin.json...
Exception validating URL: https://github.com/imDanSable/SIM/CHANGELOG (HTTP Error 404: Not Found)
FAILED
[SIM] Issues found in `plugin.json`:

Invalid URL: https://github.com/imDanSable/SIM/CHANGELOG

Fixed in 979b1241fc20525170fcbfd699358b73ea251590


Deploy Multicall3 on Oasis Network

Website: https://www.oasisprotocol.org/
Explorer: https://explorer.emerald.oasis.dev/
I already funded the deployer account.

Done, thanks!


train error please answer

Traceback (most recent call last):
File "D:\deeptrack\DCPT\lib\train../..\lib\train\trainers\base_trainer.py", line 85, in train
self.train_epoch()
File "D:\deeptrack\DCPT\lib\train../..\lib\train\trainers\ltr_trainer.py", line 136, in train_epoch
self.cycle_dataset(loader)
File "D:\deeptrack\DCPT\lib\train../..\lib\train\trainers\ltr_trainer.py", line 73, in cycle_dataset
self.actor.fix_bns()
File "D:\deeptrack\DCPT\lib\train../..\lib\train\actors\DCPT.py", line 39, in fix_bns
net = self.net.module
File "D:\Software\anaconda3\envs\Pytorch_py39\lib\site-packages\torch\nn\modules\module.py", line 1269, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'DCPT' object has no attribute 'module'

Hello! Thanks for reaching out. If you're using a single GPU for training, you can modify the code to net = self.net. However, if you use multiple GPU for training, you do not need to make this change.

Hello! Thanks for reaching out. If you're using a single GPU for training, you can modify the code to net = self.net. However, if you use multiple GPU for training, you do not need to make this change.

Thanks!!!


Bug: Crash when email address added as an alias

Describe the Bug
Linter crashes if a file contains an email address as an alias
How to Reproduce
Example to reproduce issue:
---
title: John & Jane Doe
aliases: [John & Jane Doe<EMAIL_ADDRESS>(123) 456-7890]
cssclass: [table-lines-bright, col-lines, row-lines]
---

# John & Jane Doe

This stuff doesn't really matter.

Please make sure to include the rule(s) that cause the issue if applicable. If you are not sure which rule(s) may cause the issue, please attach the data.json file located at <vault>/.obsidian/plugins/obsidian-linter/.
Expected Behavior
I expect lint4er not to crash.
Screenshots

Device

[x] Desktop
[ ] Mobile

Additional Context
Attached data.json.txt
This may be a problem with compatibility with meta-edit.
data.json.txt

Thanks for reporting this. I will have to take a closer look and see what is causing this. It is likely trying to determine what to do with the email link.

It looks like I have a fix for this. I am not sure why position would come back as undefined for the link, but with the fix in place it seems to ignore it. I guess I will wait on another bug or two before creating another release. I should have the fix on master soon.

This issue should be fixed on master. Please feel free to let us know if it is not fixed on master or in the next release.

Thank you!


Jobs run half as often as they should

After upgrading sidekiq to > 4.0 and sidekiq-cron to > 0.3.1, all of our scheduled jobs executed at exactly or almost exactly half of the frequency that they should have. For example, a job that runs once per minute would simply run once every two minutes. I'm not sure what the cause of this is, but reverting back to 0.3.1 with sidekiq 3.5.4 fixed it.

We also are missing jobs after upgrade.

I'm seeing jobs that are set to run ones a day, like 0 15 * * * running once an hour.

Any idea on this @ondrejbartas? It's stopping us from upgrading to Sidekiq 4 which has some really big improvements.

Any idea on this @ondrejbartas? It's stopping us from upgrading to Sidekiq 4 which has some really big improvements.

Any idea on this @ondrejbartas? It's stopping us from upgrading to Sidekiq 4 which has some really big improvements.

@ondrejbartas we are still eagerly waiting a response for almost a month on this :(

Same issue here

Anything new about that?

@phyzalis I gave up waiting and was looking at https://github.com/moove-it/sidekiq-scheduler but this seems to run 2x of what it should so we're just currently stuck using Sidekiq 3 still :(

I will have a look thank you. I need to be able to add jobs programmatically and this sidekiq-cron was perfect. I will have a look inside sidekiq-cron source too when I'll have time.
Thanks

For me the issue comes from RufusScheduler here:
 Rufus::Scheduler::CronLine.new(@cron).previous_time(now)

https://github.com/ondrejbartas/sidekiq-cron/blob/master/lib/sidekiq/cron/job.rb#L443
I will continue to investigate

@phyzalis Can you elaborate on that? I see that RufusScheduler supports extensions to the standard cron format, but it also claims to support standard cron format.

In fact my issue was my cron format, I was having 6 elements in it instead of 5.
Fixing this it works perfectly
For example to run something every day at 10, I was using:
0 10 * * * *
Which is wrong and run my task every hour at 10 minutes
The right way was : 0 10 * * *
But I don't think this will help you

Yeah, we were using standard 5 element cron.

@danarnold Hi sorry for delay. How can I help you? I am not really sure what is your problem.

That's fine. I'm not sure where you're confused, I described the problem in the my first comment on the issue. Is there anything I can clarify? Essentially, jobs are running 1/2 as often as they should.

@danarnold Can you give us the exact versions of the gems you are running? I created a dirt simple gem which just scheduled a job to run every minute and reports the amount of time between each invocation but can't reproduce the issue. Also it would be helpful to get your sidekiq/sidekiq-cron config.
I wrote this example worker, which although poorly written, provides a reasonable approximation to how often the cron task is running. https://gist.github.com/hderms/bf484c103573f65942f78cff5243d845

This was with sidekiq 4.0.1, sidekiq-cron 0.4.2.
Here's our cron file with the worker names changed (they are all unique in our app, of course):
redacted_worker:
  cron: '*/10 * * * *'
  class: 'RedactedWorker'
  queue: low

redacted_worker:
  cron: '*/1 * * * *'
  class: 'RedactedWorker'
  queue: slow

redacted_worker:
  cron: '*/1 * * * *'
  class: 'RedactedWorker'
  queue: slow

redacted_worker:
  cron: '*/1 * * * *'
  class: 'RedactedWorker'
  queue: slow

redacted_worker:
  cron: '*/5 * * * *'
  class: 'RedactedWorker'
  queue: low

redacted_worker:
  cron: '0 * * * *'
  class: 'RedactedWorker'
  queue: slow

redacted_worker:
  cron: '0 0 */1 * *'
  class: 'RedactedWorker'
  queue: slow

redacted_worker:
  cron: '0 */2 * * *'
  class: 'RedactedWorker'
  queue:  slow

redacted_worker:
  cron: '0 */1 * * *'
  class: 'RedactedWorker'
  queue: low

redacted_worker:
  cron: '*/5 * * * *'
  class: 'RedactedWorker'
  queue: low

redacted_worker:
  cron: '*/5 * * * *'
  class: 'RedactedWorker'
  queue: low

It's worth mentioning that we run these workers in production on Heroku with multiple sidekiq instances, so Heroku fires up at least one dyno per queue, meaning that you get dyno processes running sidekiq -q slow, running sidekiq -q low, etc. This doesn't cause issues using the version we're using right now (sidekiq-cron 0.3.1, sidekiq 3.5.4).

Thanks @danarnold I will see if I can craft a minimal example project which reproduces the issue.

@danarnold can you reproduce this issue on your development machine or has it only been encountered on heroku?

I didn't try, because I had it happening on Heroku. If need be I could try to reproduce it on my machine.

@hderms, @danarnold and I work together and I can confirm it happens in both our staging and my local environment.

@jclusso good to know. I tried crafting a repo with the bare minimum setup
(sidekiq-cron,  multiple cron workers registered in YAML file and running
multiple sidekiq processes) but can't reproduce the issue. Are there any
other quirks of the environment you think might be important offhand?
On Apr 1, 2016 12:42 AM, "Jarrett Lusso"<EMAIL_ADDRESS>wrote:

@hderms https://github.com/hderms, @danarnold
https://github.com/danarnold and I work together and I can confirm it
happens in both our staging and my local environment.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/ondrejbartas/sidekiq-cron/issues/89#issuecomment-204255342

@hderms I can't think of any quirks. I'd love to hear @gdeglin chime in since he reported the same problem. I've also attached our sidekiq config.
host = ENV['redis_host']
port = ENV['redis_port']
db = ENV['redis_db_sidekiq']
namespace = ENV['redis_namespace']

if Rails.env.development? || Rails.env.test?
  redis_url = "redis://#{host}:#{port}/#{db}"
else
  password = ENV['redis_password']
  redis_url = "redis://:#{password}@#{host}:#{port}/#{db}"
end

Sidekiq.configure_server do |config|
  config.redis = { url: redis_url, namespace: namespace }
  schedule_file = 'config/schedule.yml'

  if File.exist?(schedule_file)
    Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)[Rails.env]
  end
end

Sidekiq.configure_client do |config|
  config.redis = { url: redis_url, namespace: namespace }
end

Nothing unusual here either. We have a task that should run every 5 minutes, and once ever 4 hours or so it would skip over one of the scheduled times. My only theory is that it could be NTPd related, but I haven't had a chance to try disabling NTPd to see if it fixes it.

@jclusso is @gdeglin sharing info which is consistent with how the issue presents in your experience i.e. once every X amount of time there is a deviation or is it presenting differently. I'm concerned about this issue but can't figure out how to reproduce it. My limited example app is available at: https://gist.github.com/hderms/b769dc573394db61d52531b464de4980

@hderms we will need to to do further analysis on the pattern to see if that is the case. From what we see it appears to just be doing half the number of workers.

I don't believe it correlates with that pattern at all. It is a simple 1/2 jobs.

Have you tried to setup: average_scheduled_poll_interval or poll_interval in sidekiq configuration?

@ondrejbartas I'm not sure what that would do. We have it set to the sidekiq default, which is approximately every 15 seconds balanced over the number of workers that are running. I see no reason to change it.

@ondrejbartas we upgraded sidekiq at my current workplace and our jobs are running at the specified frequency. The evidence right now, both my experience with my work's production codebase and my stripped down sample sidekiq project provided earlier in this thread, seems to suggest there is something unusual going on that we can't seem to capture without more information. If you guys can reproduce a minimal project which has jobs running at half frequency that would be extremely helpful in debugging.

I'm having a possible related issue? My jobs are running mostly fine but every so often they will simply not be added to the queue. Most of my jobs run once a day and I've noticed about twice this month they simply did not run and the sidekiq log shows no errors (it wasn't even added!). I've since added a job that runs every minute and every so often I can see that it simply skips a minute and doesn't even run. Going through my own logs and audits it looks like this issue present itself around March of 2016 for me which is precisely when I updated to Sidekiq (4.1.1) and Sidekiq-cron (0.4.2)
Before this there is no evidence of any issues and I was running Sidekiq (3.3.4) and Sidekiq-cron (0.3.1).
I'm going to revert one of my clients back to these versions and see if the issue disappears.

By the way our issues ended up being long running workers getting shutdown before completion by our autoscaling of Heroku dynos. We ended up implementing locking mechanisms. We also stopped autoscaling worker queues.

Just realized I never came back with what happened.
Reverting Sidekiq and Sidekiq-cron back fixed my issues. Not sure if both or only one of them was causing the problem but I'm not willing to test this in production at the moment since everything is now back to normal.

Hi i had a similar problem. Using default sidekiq config poll_interval_average: nil, average_scheduled_poll_interval: 15 which with one running process (just for the cron) it defaults to process_count *  average_scheduled_poll_interval = 15 seconds. Yet I observed that sidekiq-cron is checking cron-table every 60+ seconds, like minute and 5s, or minute and 10s.
And that's really strange, because that app's only job is to clear some keys from redis every day at midnight, and provide web gui for other apps using sidekiq and this redis instance. So basically most of the time it sits there idle.
I don't know why this happens, wasn't able to track it down. But since the execution time window for job to be started according to cron is time from cron-table + 60s and with pooling interval a slightly more than 60s you can sometimes miss the job execution. My simple naive solution was simply removing that 60s execution window https://github.com/sztosz/sidekiq-cron/commit/9d2eb40fa674e8ec4e22720499275157647b040d
@ondrejbartas I was curious if you can help with tracking what in sidekiq-cron can make checking cron-table so rarely (60+ seconds) instead of default 15 sec of pooling interval. Or maybe if there is a workaround other than me deleting that one particular line (link in paragraph above)?

@ondrejbartas We are running sidekiq pro 4.1.1 and sidekiq-cron 0.4.3 and our scheduled sidekiq-cron jobs are randomly not being enqueued. An important and time sensitive job didn't run on Sunday which made us aware of the situation, but after further research, there are more jobs that are not enqueueing consistently. Is there any progress on this issue?  The changelog looks like it is not up to date, so I don't know what 0.4.1, 0.4.2, and 0.4.3 changed.
Thanks!
Teresa

#153 #150 #152

released 0.6.0


[Improvement]: Use CachingCatalog to reduce the time cost of IcebergCatalogWrapper#loadTable

Search before asking

[X] I have searched in the issues and found no similar issues.

What would you like to be improved?
Every time ArcticTable is obtained, it will go through a complete load table process(org.apache.iceberg.catalog.Catalog#loadTable). We can cache the loaded table to reduce the time cost of get ArcticTable.
How should we improve?
import org.apache.iceberg.CachingCatalog
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Subtasks
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

I am more concerned about how to obtain the latest state if there is caching.

I am more concerned about how to obtain the latest state if there is caching.
Via refresh() to obtain the latest state?

I think TableRuntimeRefreshExecutor must use the fresh table,  Cache tables can be used as appropriate in other areas.

I believe that adding CacheCatalog can indeed reduce frequent access to the metadata system in some scenarios where tables need to be frequently loaded, such as frequent read operations on Dashboard.
However, if a refresh is performed every time a table is loaded, it seems that the overhead reduction may not be much compared to the cost of loading the table each time.
If this is true, we may need to determine where to force table refresh and manually refresh when needed.


cannot schedule new futures after interpreter shutdown when use cloud (s3) to upload outputs

**app.py
import os
import pathlib
import ffmpeg_streaming
from ffmpeg_streaming import Formats, S3, CloudManager

BASE_DIR = str(pathlib.Path(__file__).parent)

INPUT_PATH = os.path.join(BASE_DIR, 'video.mp4')
OUTPUT_PATH = os.path.join(BASE_DIR, 'output')

video = ffmpeg_streaming.input(INPUT_PATH)
hls = video.hls(Formats.h264())
hls.auto_generate_representations()

s3 = S3(
    region_name='sgp1',
    aws_access_key_id='<hidden_for_security_reason>',
    aws_secret_access_key='<hidden_for_security_reason>'
)

save_to_s3 = CloudManager().add(s3, bucket_name="tokbee")

hls.output(clouds=save_to_s3)

** Error Log
app_1  | Error in atexit._run_exitfuncs:
app_1  | Traceback (most recent call last):
app_1  |   File "/usr/local/lib/python3.8/site-packages/ffmpeg_streaming/_clouds.py", line 59, in upload_directory
app_1  |     self.s3.upload_file(join(directory, file), bucket_name, join(folder, file).replace("\\", "/"))
app_1  |   File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 143, in upload_file
app_1  |     return transfer.upload_file(
app_1  |   File "/usr/local/lib/python3.8/site-packages/boto3/s3/transfer.py", line 284, in upload_file
app_1  |     future = self._manager.upload(
app_1  |   File "/usr/local/lib/python3.8/site-packages/s3transfer/manager.py", line 330, in upload
app_1  |     return self._submit_transfer(
app_1  |   File "/usr/local/lib/python3.8/site-packages/s3transfer/manager.py", line 525, in _submit_transfer
app_1  |     self._submission_executor.submit(
app_1  |   File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 474, in submit
app_1  |     future = ExecutorFuture(self._executor.submit(task))
app_1  |   File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 181, in submit
app_1  |     raise RuntimeError('cannot schedule new futures after '
app_1  | RuntimeError: cannot schedule new futures after interpreter shutdown
hls_transcoder_pro_app_1 exited with code 0

I am also facing the same issue!!
Can anyone help me?


Why is callsPerHost attribute of AsyncCall class not static?

Now we need to copy it from pre-call to new call which will enqueue readyAsyncCalls collection. This callsPerHost attribute of AsyncCall always is abandoned when the object enqueue.
 val existingCall = findExistingCallWithHost(call.host())
        if (existingCall != null) call.reuseCallsPerHostFrom(existingCall)

internal inner class AsyncCall(
    private val responseCallback: Callback
  ) : Runnable {
    @Volatile private var callsPerHost = AtomicInteger(0) // why not static
    fun callsPerHost(): AtomicInteger = callsPerHost

    fun reuseCallsPerHostFrom(other: AsyncCall) {
      this.callsPerHost = other.callsPerHost // original object will be abandoned
    }
... ...

Because you can have multiple OkHttpClient instances in an application.

In general, if you have questions like this consider changing the code and running our test suite. That can be illuminating.


error when hitting generate?

I had a bit of trouble setting this up, but now it appears to be working until I hit the generate button.
Error message.

Then this error message shows up on the webpage itself.

It seems like the ajax call in bundle.js is going to something that doesn't exist on line 204. I have downloaded various versions of audiogram and can't find any that have a 'submit' directory.
$.ajax({
		url: "/submit/",
		type: "POST",

Any ideas? I work for the Atlanta Journal and would like to use this for multiple reporters. I am using a remote server running Ubuntu 14.04 and can provide the URL is someone is willing to help.
THANKS,
George Mathis

There is no submit directory, it's a virtual path defined in the server files. What that error message suggests to me is that the problem is some combination of:
-You're running nginx as a proxy in front of Audiogram, and that's misconfigured
-Audiogram is running on port 80 and you also have another webserver running on the same port, so that's getting tangled up
Possible remedies:
-Running Audiogram on its default port (8888) and visiting it that way (e.g. http://yourdomain:8888/)
-Running it on a separate remote server that's not shared with other web stuff
Let me know if you can't resolve it and I'll try to dig into it some more.

Thanks for the quick reply. I have tried various things, such as retooling the nginx setup as per Node.js documentation and I am still getting the same error. (I followed this guide for node and nginx cohabitation.)
I have PM2 running the server and worker and the page is being served up fine, but it seems the paths provided in editor/index (to bundle.js and css) and the paths provided by bundle.js (to theme.json and the background images ... and I assume the SUBMIT function) are returning relative paths, not absolute ones.
I am surely doing something wrong, but I followed the Ubuntu 14+ directions.
Not sure if you work with nginx much but my conf file looks like this ...
server {
listen 80;
#server_name ec2-54-224-152-237.compute-1.amazonaws.com;
server_name <IP_ADDRESS>;

location = /george/audiogram/ {
        proxy_pass http://<IP_ADDRESS>:8888/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
...

It seems to be serving the html up fine, though I have replaced the relative paths in editor/index.html with absolute paths (otherwise nothing works)
Thanks for any assistance. Please excuse my node.js ignorance ... I am not familiar with it, obviously.
I can always look into getting a pure node js instance, but it seems like this should work with an nginx proxy.

Unfortunately I can't really help with the nginx side of things, but that does seem to be squarely where the issue is - specifically regarding the relative vs. absolute paths.


CDSData streams should be synced on initialization

CDSData streams which include the various draw tool streams should sync the data in the data source on initialization. Currently the .data on these streams isn't updated until the first change event on the data source is triggered, instead it should update the data immediately after the plot is constructed.

This seems like a good policy to me.


Support KUBECONF env variable

I can't agree. Let's see following functions read my comments:
func shouldUseContainerMode() (bool, error) {
	// When kube config is set, container mode is not used
	if os.Getenv(k8sutil.KubeConfigEnvVar) != "" {
		return false, nil                             --- My case - return false
	}

func initialize(kubeconfig string) {
	if kubeconfig == "" {
		// skip out-of-cluster initialization if inside the container
		if kc, err := shouldUseContainerMode(); kc && err == nil {       --- kc == false -> continue
			return
		} else if err != nil {
			logrus.Errorf("could not determine if running in a container: %v", err)
		}
		var err error
		kubeconfig, err = getDefaultKubeConfigFile()      --- Load kubernetes config from HOME dir (not from KUBECONF)
		if err != nil {
			panic(err)
		}
	}
	os.Setenv(k8sutil.KubeConfigEnvVar, kubeconfig)           --- Override my KUBECONF property
}

Source:
https://github.com/apache/camel-k/blob/21eb85be58b3bb8807180bc24892c17adb7b12df/pkg/client/client.go#L204-L208
https://github.com/apache/camel-k/blob/master/pkg/client/client.go#L136-L150

The problem is that the code should even not enter that if branch, because the kubeconfig variable should have been populated with the KUBECONF env var or with the value passed via --config flag.
I'd like to understand why it isn't and fix that.

Ok, I am trying to run e2e tests. I forget to mention it. Test client is initialized with empty string.
func newTestClient() (client.Client, error) {
	return client.NewOutOfClusterClient("")
}

https://github.com/apache/camel-k/blob/master/e2e/test_support.go#L82

To avoid inconsistencies between --config and KUBECONF, what do you think about moving all references to os.getEnv outside the pkg/client package and rely on the kubeconfig variable only.
Then we change the tests to pick values from the KUBECONF env (cobra commands already initializes it from env or from the --config flag).

Sounds reasonable. I will update my PR on Monday.

@nicolaferraro you are right. https://github.com/apache/camel-k/blob/master/pkg/cmd/root.go#L54 should be enough. Please check my test-fix.


Python error code conformance

Error codes generated in python wrapping code should comply with the spec in "gRPC Error Normalization" spreadsheet.

We're way out of compliance, but I've added a "Python" column for the time being describing what we surface.
Retitling this issue to reflect the work to be done bringing us to conformance.

anything missing here?

The important parts of this are fixed as of 1d750d6ca2d878385e4a.
I think we would still have a problem if there's a serialization or deserialization error for protocol buffer objects, but that's deferrable.

Fixed by recent GA changes.


listCiphers vs getCiphers

Hey, awesome project!
Is there a reason the name is listCiphers() in this module and not the same as in node.js crypto.getCiphers() ?
Maybe keep both for backwards compatibility, but it's surprising that getHashes is there and getCiphers is not...

that is a bug, see calvinmetcalf/browserify-aes#15

great!


New branch

The link to the video is below.
https://www.loom.com/share/3a48a54920824a8ab02317b799f373b8
please give a detailed feedback so I can fix the issues if there are one I would be very thankful of you for your kind feedback.
created featured speakers section dynamically. But named it as Featured Heros
as you had mentioned I have created another video for this this link for that is below.
https://www.loom.com/share/6d44b593083d40eb9d8ec407081bf893

Hi @shobeee,
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉
Highlights

The Pull Request has a proper title and description. ✔
You have added a descriptive Readme file. Good Job. ✔
There is no linter error is present. ✔
Project has met all the technical aspects you needed for this project. Keep rocking! ✔

Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


add an option to export pyarrow record batch

it seems daft support only exporting arrow table, which materialized the data into memory,  I have a scenario  where I run daft in an environment with limited RAM and adding support for record batch will be useful as the data is streamed in a smaller batch
thanks

@djouallah you should be able to iterate over the partitions to get a similar effect
df = daft.read_parquet('/path/to/files')

for partition in df.iter_partitions():
    tbl = partition.to_arrow()
    # do something with tbl

But I agree, a native to_recordbatches may be more intuitive

please don't close a bug report, unless it is confirmed that it fixes the issue
does not work for my use case
df = df.with_column('DATE', col('SETTLEMENTDATE').cast(DataType.date()))
	df = df.with_column('year', col('SETTLEMENTDATE').dt.year())
	write_deltalake(f"/lakehouse/default/Tables/T{total_files}/daft",df.to_arrow_iter(),schema= df.schema().to_pyarrow_schema(), mode="append", partition_by=['year'], storage_options= storage_options)

getting this error
File ~/jupyter-env/python3.10/lib/python3.10/site-packages/pyarrow/dataset.py:1030, in write_dataset(data, base_dir, basename_template, format, partitioning, partitioning_flavor, schema, filesystem, file_options, use_threads, max_partitions, max_open_files, max_rows_per_file, min_rows_per_group, max_rows_per_group, file_visitor, existing_data_behavior, create_dir)
   1027         raise ValueError("Cannot specify a schema when writing a Scanner")
   1028     scanner = data
-> 1030 _filesystemdataset_write(
   1031     scanner, base_dir, basename_template, filesystem, partitioning,
   1032     file_options, max_partitions, file_visitor, existing_data_behavior,
   1033     max_open_files, max_rows_per_file,
   1034     min_rows_per_group, max_rows_per_group, create_dir
   1035 )

File ~/jupyter-env/python3.10/lib/python3.10/site-packages/pyarrow/_dataset.pyx:4010, in pyarrow._dataset._filesystemdataset_write()

File ~/jupyter-env/python3.10/lib/python3.10/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowTypeError: Could not unwrap RecordBatch from Python object of type 'pyarrow.lib.Table'

@jaychia I am doing a presentation in a two months and really would like to show a different story,

please don't close a bug report, unless it is confirmed that it fixes the issue
does not work for my use case
df = df.with_column('DATE', col('SETTLEMENTDATE').cast(DataType.date()))
df = df.with_column('year', col('SETTLEMENTDATE').dt.year())
write_deltalake(f"/lakehouse/default/Tables/T10/daft",df.to_arrow_iter(),schema= df.schema().to_pyarrow_schema(), mode="append", partition_by=['year'], storage_options= storage_options)

getting this error
File ~/jupyter-env/python3.10/lib/python3.10/site-packages/pyarrow/dataset.py:1030, in write_dataset(data, base_dir, basename_template, format, partitioning, partitioning_flavor, schema, filesystem, file_options, use_threads, max_partitions, max_open_files, max_rows_per_file, min_rows_per_group, max_rows_per_group, file_visitor, existing_data_behavior, create_dir)
   1027         raise ValueError("Cannot specify a schema when writing a Scanner")
   1028     scanner = data
-> 1030 _filesystemdataset_write(
   1031     scanner, base_dir, basename_template, filesystem, partitioning,
   1032     file_options, max_partitions, file_visitor, existing_data_behavior,
   1033     max_open_files, max_rows_per_file,
   1034     min_rows_per_group, max_rows_per_group, create_dir
   1035 )

File ~/jupyter-env/python3.10/lib/python3.10/site-packages/pyarrow/_dataset.pyx:4010, in pyarrow._dataset._filesystemdataset_write()

File ~/jupyter-env/python3.10/lib/python3.10/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowTypeError: Could not unwrap RecordBatch from Python object of type 'pyarrow.lib.Table'

Looks like our .to_arrow_iter() API currently returns an Arrow Table rather than a RecordBatch.
@djouallah here's a way you can convert the Tables into a RecordBatches instead for a quick workaround. I'll look into a way to make this native to Daft tomorrow.
import daft
from daft import DataType, col

df = daft.from_pydict({"SETTLEMENTDATE": ["2021-01-01", "2021-01-01", "2021-01-01"]})
df = df.with_column('DATE', col('SETTLEMENTDATE').cast(DataType.date()))
df = df.with_column('year', col('DATE').dt.year())

# Convert the tables into record batches as a streaming iterator
arrow_batches_iter = (batch for tbl in arrow_iter for batch in tbl.to_batches())

next(arrow_batches_iter)
Project-Project [Stage:1]:   0%|                                                                                | 0/1 [00:00<?, ?it/s]Out[9]:
pyarrow.RecordBatch
SETTLEMENTDATE: large_string
DATE: date32[day]
year: int32
----
SETTLEMENTDATE: ["2021-01-01","2021-01-01","2021-01-01"]
DATE: [2021-01-01,2021-01-01,2021-01-01]
year: [2021,2021,2021]

It seems like returning RecordBatch should be the more canonical API here, it was our mistake that we returned Tables. We'll rectify that, and likely provide alternative .to_arrow_record_batches()/.to_arrow_tables() APIs. Let me check with the team to see what they think tomorrow 🙂

Hi @djouallah,
this should be in the next release!

thank you, it is working,