Translate IOU buttons

Details
Makes sure that the IOU confirmation button is always translated to the selected language
Fixed Issues
$ https://github.com/Expensify/App/issues/14305
Tests

Open the App in 2 sessions logged in to the same account
In session A, navigate to + > Send money > Enter Amount > Select participant`
Note the text of the confirmation button
In session B, navigate to Settings > Preferences and change the selected language
In session A, verify that the confirmation button was translated accordingly
In session A, select a different option in the dropdown button and repeat steps 4-5
Repeat steps 2-6 for the following pages:

+ > Request money > Enter amount > Select participant

+ > Split bill > Enter amount > Select participants

Select an open IOU where you owe money

[x] Verify that no errors appear in the JS console

Offline tests

QA Steps

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://user-images.githubusercontent.com/22219519/212781348-7b0f8631-aa38-427b-b0b0-953978d72e2b.mov

Mobile Web - Chrome
https://user-images.githubusercontent.com/22219519/212781357-e3842114-094a-451d-9e19-7f66468acb97.mov

Mobile Web - Safari
https://user-images.githubusercontent.com/22219519/212781366-f3fe14ec-52b9-4b2c-845f-67829cbab647.mov

Desktop
https://user-images.githubusercontent.com/22219519/212781374-c924dbdf-2596-475f-abc8-37b1a041bd24.mov

iOS
https://user-images.githubusercontent.com/22219519/212781386-2375fd29-efe3-425e-b4d4-95caf47e025c.mov

Android
https://user-images.githubusercontent.com/22219519/212781392-2e305035-aee9-4a1a-969e-eb37242f84ed.mov

Reviewer Checklist

[ ] I have verified the author checklist is complete (all boxes are checked off).
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the steps for local testing are in the Tests section
[ ] I verified the steps for Staging and/or Production testing are in the QA steps section
[ ] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I included screenshots or videos for tests on all platforms
[ ] I verified tests pass on all platforms & I tested again on:

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[ ] MacOS / Chrome / Safari
[ ] MacOS / Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[ ] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[ ] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines
[ ] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If any new file was added I verified that:

[ ] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://user-images.githubusercontent.com/85645967/213007881-37ed3d2f-dd99-43de-9bab-22ffa3af9ea3.mov

Desktop
https://user-images.githubusercontent.com/85645967/213007831-3c337010-cc11-4e8b-b78c-0c445651039f.mov

Mobile Web - Chrome  & Mobile Web - Safari
https://user-images.githubusercontent.com/85645967/213008709-5dd9eefb-37c0-439c-8a63-427787a78cde.mov

iOS

Android

🚀 Deployed to staging by https://github.com/youssef-lr in version: 1.2.57-2 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/luacmartins in version: 1.2.57-3 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


How to disable watchtower on a single container?

The Readme doesn't answer this question directly. I assume setting com.centurylinklabs.watchtower.enable to false will disable updates for a single container? Do I need to change something on my start parameter?

Thanks for your report. In order to use the label you have to pass --label-enable after having passed this flag, only containers with the label com.centurylinklabs.watchtower.enable set to true will be updated.

Is there a way to make the exact opposite? I'd like to auto-update all containers except a single one.

Exactly what I need for my comment #167 . Unfortunately, it seems there is no safe way to do this.

It would be possible to change the behaviour, that containers which have the com.centurylinklabs.watchtower.enable set to false are always excluded from the updates. Those would be also excluded if the --label-enable wasn't specified.
@redge76 would that be a possible trade-off for you?

Yeah, I know it's not the best solution. But it would be a trade-off where we don't break anything for the people out there but at least give some safer behaviour.

Thanks!

@Skaronator you're welcome. Please note a new release is scheduled for in about one week.

Hi,
Still no release ?

please release this. My logs are full of spam because of some containers that watchtower wants to update, but fails to do so.


[Stremio] Any way to set the default IP?

Hey, I'm running this locally and both ends individually work fine.
The WebUI has a default IP of <IP_ADDRESS> but that doesn't work out the box. Only when I set the IP to that of the machine running this image (192.168.0.XX) does it then seem to work. I would assume <IP_ADDRESS> only works when the machine you are accessing the WebUI from is also the machine running the container.
Is there a way to set the default server IP, so that my friends on the same network do not have to each set the IP of the server the first time they connect?
Thanks :)

Hey!
Unfortunately, I don't think so :(
I think the default server url is defined in stremio-core package.
https://github1s.com/Stremio/stremio-core/blob/HEAD/src/constants.rs#L47-L48
You can rebuild stremio-web with a custom version of stremio-core, but it's not easily portable.
Another way is to hot patch the WebAssembly binary (web/binaries/stremio_core_web_bg.wasm). I can check this.

Hey @danperks!
After some research, I found that the streamingServer query component isn't implemented on stremio-web (reference).
So I added a local copy of the staging stremio-shell Web UI (https://staging.strem.io/) accessible on port 8081 (see https://github.com/n0bodysec/docker-images/commit/777414624c9dbb31f7c9c440f258fdd085dcc0dc).
So now, you can reach the old webui (the one that supports the streaminServer query component) at: http://stremio:8081
Example usage: http://stremio:8081/?streamingServer=http://stremio:11470
I hope this solves your problem!

I'll give it a try later on, thanks for the help.

Is there a way to do this for the new Web-UI, for example by patching the web assembly? Or is this only supported by the old UI? The query parameter does not seem to work on the new one

Is there a way to do this for the new Web-UI, for example by patching the web assembly? Or is this only supported by the old UI? The query parameter does not seem to work on the new one

The query component is not implemented on the new UI. I tried to patch the WebAssembly binary (see https://github.com/n0bodysec/docker-images/issues/4#issuecomment-1266086307), but the binary gets corrupted.
Btw I like the idea you mention in https://github.com/n0bodysec/docker-images/issues/5#issuecomment-1299983762.


Update fr.txt

Translate to french the todo lines. Not sure what to do for this : # TODO : add an option to translate "Untitled" by "Sans nom". Correct some tabs.

Looks good, thank you!


Update CaseInsensitive table to Unicode 9.0

ECMA 262 specifies implementations should use the latest Unicode standard.
As of time of writing, that is Unicode 9.0.
Related to #517, #2356

Removing "Bug" as this is not a regression, but is still a work item to follow up.


Decide on a good interchange format for 3D structures

Besides the 3D structure, we'll also want to hold on to partial charges and other information.
You can store that in CIFs (which are annoying), extXYZ, or - even - Pandas DataFrames.

Yea, it is. I think we've settled on VASP POSCARs for 3D structures.


(PRODUCT PAGE) А dot as a decimal separator in the product rating (filter)

Pre-conditions:

Go to product page with not empty rating

Steps:

View the product rating on the ratings filter block

ER:
А comma as a decimal separator in the product rating
AR:
А dot as a decimal separator in the product rating

Hello @ilsinyakov, can you please assign this to me? I'd like to take it up.

Hello @Akhi88S
Done


"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: kamaji.clastix.io/v1alpha1: the server could not find the requested resource"

Hi,
I installed version 0.3.0 on our test cluster using the installation method as provides in the release nodes. The capi-kamaji-controller-manager is running in the kamaji-system namespace, however, it's throwing this error:
2023-08-20T11:36:02Z    ERROR   controller-runtime.source.EventHandler  failed to get informer from cache       {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: kamaji.clastix.io/v1alpha1: the server could not find the requested resource"}
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/source/kind.go:68
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/loop.go:62
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
        /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/loop.go:63
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
        /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/source/kind.go:56

Looks to me like it's missing RBAC permissions?

Hi,
I installed version 0.3.0 on our test cluster using the installation method as provides in the release nodes. The capi-kamaji-controller-manager is running in the kamaji-system namespace, however, it's throwing this error:
2023-08-20T11:36:02Z    ERROR   controller-runtime.source.EventHandler  failed to get informer from cache       {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: kamaji.clastix.io/v1alpha1: the server could not find the requested resource"}
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/source/kind.go:68
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/loop.go:62
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
        /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/loop.go:63
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
        /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/source/kind.go:56

Looks to me like it's missing RBAC permissions?

Did you install kamaji controller in your cluster?
(getting started : https://kamaji.clastix.io/getting-started/)

I did not :sweat_smile:
That said, it's not really mentioned anywhere in this repo that I noticed. I know it kinda makes sense, now that I think about it, but having it in the docs would be nice as well.

@timstoop Unfortunately there's no way to inject dependencies using the clusterctl manifests.
Your point in increasing the documentation without giving anything for granted is valid, please, contribute to the project since it's much easier reading a markdown file rather than looking for errors output in the issues section.


Fix resource conf template

Description
openvpn_conf is unusable with a resource name different than server
Check List

[ ] All tests pass. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/TESTING.MD
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable

          1 Error

:no_entry_sign:
Please include a CHANGELOG entry.

          1 Warning

:warning:
This Pull Request is probably missing tests.

  Generated by :no_entry_sign: Danger

@gionn can you please add a ChangeLog entry?

Released as: 5.3.0


downloader fails

Traceback (most recent call last):
File "/Users/chhu02/anaconda3/bin/icloudpd", line 10, in
sys.exit(main())
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/icloudpd/base.py", line 229, in main
client_id=os.environ.get("CLIENT_ID"),
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/icloudpd/authentication.py", line 32, in authenticate
client_id=client_id)
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/pyicloud_ipd/base.py", line 193, in init
self.authenticate()
File "/Users/chhu02/anaconda3/lib/python3.6/site-packages/pyicloud_ipd/base.py", line 216, in authenticate
raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Failed to validate the credentials from cookie',))

I got the same. It seems some change on Apple side

I got the same. It seems some change on Apple side.
I can confirm this, too. Lets hope pyicloud can fix this soon. I hope I find some time this weekend to work on this as well, if still needed.

apple change login logic , resolve:

Can any one please implement in java  for https://github.com/viable-hartman/InflatableDonkey/tree/TwoFAHacky please help

url https://setup.icloud.com/ws/1
used for authenticate no longer available or not supported

apple change login logic , resolve: ...

Hi @huangbiyong ,
Can you please rephrase what you are meaning with your comment/screenshots? Do you have a fix already for this issue? If so you could send me the files so that I can review/test/checkin.
cheers,
Jörg

Can refer to my github:   https://github.com/huangbiyong/Apple-iCloud-lastest-API

Hey!
I've managed to make it work. You should get this version of pyicloud:
https://github.com/PeterHedley94/pyicloud
and then you should comment out these lines from icloudpd/base.py:
# if skip_videos and photo.item_type != "image":
#     logger.set_tqdm_description(
#         "Skipping %s, only downloading photos." % photo.filename
#     )
#     break
# if photo.item_type != "image" and photo.item_type != "movie":
#     logger.set_tqdm_description(
#         "Skipping %s, only downloading photos and videos. "
#         "(Item type was: %s)" % (photo.filename, photo.item_type)
#     )
#     break
It works now, it is downloading photos perfectly.

Noob question.  How can I replace my current version of pyicloud with this version https://github.com/PeterHedley94/pyicloud?
Thank you.

Hi @huangbiyong ,
I cloned the repository and run sudo python setup.py install in the pyicloud directory. I also I commented out the mentioned lines.

        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break

But I still get the "same" error message:
2019-06-18 19:14:31 DEBUG    Authenticating...
iCloud Password:
Traceback (most recent call last):
  File "/usr/local/bin/icloudpd", line 11, in <module>
    load_entry_point('icloudpd==1.4.3', 'console_scripts', 'icloudpd')()
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/icloudpd/base.py", line 229, in main
  File "build/bdist.linux-x86_64/egg/icloudpd/authentication.py", line 39, in authenticate
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.9.9-py2.7.egg/pyicloud_ipd/base.py", line 193, in __init__
    self.authenticate()
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.9.9-py2.7.egg/pyicloud_ipd/base.py", line 216, in authenticate
    raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError(u'Failed to validate the credentials from cookie',))
2019-06-18 19:14:34 DEBUG    Authenticating...
Traceback (most recent call last):
  File "/usr/local/bin/icloudpd", line 11, in <module>
    load_entry_point('icloudpd==1.4.3', 'console_scripts', 'icloudpd')()
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/icloudpd/base.py", line 229, in main
  File "build/bdist.linux-x86_64/egg/icloudpd/authentication.py", line 32, in authenticate
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.9.9-py2.7.egg/pyicloud_ipd/base.py", line 193, in __init__
    self.authenticate()
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.9.9-py2.7.egg/pyicloud_ipd/base.py", line 216, in authenticate
    raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError(u'Failed to validate the credentials from cookie',))

Do you have any Idea what went wrong and could help me with that?

I also have problems with icloudpd, does anyone have any working workarounds?

@danielfmolnar 's fix does it work? If so can anyone build a new docker image for it so I can update my icloud-sync job

Hi everyone, sorry I don't have any time to investigate this issue or work on a fix, but I'm very happy to merge a PR and release a new version + docker image. Thanks!

It looks like we might need a change in the pyicloud library, so here's my fork of pyicloud that icloud_photos_downloader uses (also I use the pyicloud-ipd branch in this repo.)
If someone could please send a PR to update the pyicloud-ipd branch for my pyicloud fork with the required changes, and then make sure that the test suite is passing for icloud_photos_downloader. Then I'm happy to push out the new versions

@nbroadbent
Hey!
I have forked both libraries. They work. I merged the branch from peterheadley94.

So, apparently his code managed to fix the authentication issue, but it did not work with your photo downloader library. I had to uncomment a paragraph. Now it works perfectly like it used to.

hi, I'm not really good with pip and git, can u say me how to fix it pls

@danielfmolnar can you do a PR for this to https://github.com/ndbroadbent/pyicloud/tree/pyicloud-ipd ? Or at least post diffs of the changes to this thread? Thanks...

Hey!
I've managed to make it work. You should get this version of pyicloud
https://github.com/PeterHedley94/pyicloud
and then you should comment out these lines from icloudpd/base.py:
        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break

These lines have to be commented out because otherwise I get the error that the PhotoAsset object has no attribute called "item_type".
It works now, it is downloading photos perfectly.

I cloned that version of pyicloud and it doesn't work for me;
Traceback (most recent call last):   File "/usr/local/bin/icloud", line 11, in <module>     load_entry_point('pyicloud==0.9.1', 'console_scripts', 'icloud')()   File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/cmdline.py", line 202, in main     password.strip()   File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 194, in __init__     self.authenticate()   File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 208, in authenticate     myICloud = hack.PyiCloudService()   File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/hack.py", line 232, in __init__     super(PyiCloudService, self).__init__(self) TypeError: super() argument 1 must be type, not classobj

Hey!
I've managed to make it work. You should get this version of pyicloud
https://github.com/PeterHedley94/pyicloud
and then you should comment out these lines from icloudpd/base.py:
        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break

These lines have to be commented out because otherwise I get the error that the PhotoAsset object has no attribute called "item_type".
It works now, it is downloading photos perfectly.

I cloned that version of pyicloud and it doesn't work for me;
Traceback (most recent call last): File "/usr/local/bin/icloud", line 11, in <module> load_entry_point('pyicloud==0.9.1', 'console_scripts', 'icloud')() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/cmdline.py", line 202, in main password.strip() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 194, in __init__ self.authenticate() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 208, in authenticate myICloud = hack.PyiCloudService() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/hack.py", line 232, in __init__ super(PyiCloudService, self).__init__(self) TypeError: super() argument 1 must be type, not classobj

Hey!
You should clone my repo, that version should work. Actually, PeterHedley94's version in itself did not work for me either, that's why I had to merge his changes into ndbroadbent's pyicloud library. Both of them are necessary because ndbroadbent's repo is fully compatible with his photo downloader code and  PeterHedley94's changes (adding hack.py) solves the authentication issue.
So I recommend that you pull my forks:
https://github.com/danielfmolnar/icloud_photos_downloader
https://github.com/danielfmolnar/pyicloud
Please let me know if they don't work properly.

@danielfmolnar can you do a PR for this to https://github.com/ndbroadbent/pyicloud/tree/pyicloud-ipd ? Or at least post diffs of the changes to this thread? Thanks...

Hey!
I've created the pull requests for both repos.

Hey!
I've managed to make it work. You should get this version of pyicloud
https://github.com/PeterHedley94/pyicloud
and then you should comment out these lines from icloudpd/base.py:
        # if skip_videos and photo.item_type != "image":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos." % photo.filename
        #     )
        #     break
        # if photo.item_type != "image" and photo.item_type != "movie":
        #     logger.set_tqdm_description(
        #         "Skipping %s, only downloading photos and videos. "
        #         "(Item type was: %s)" % (photo.filename, photo.item_type)
        #     )
        #     break

These lines have to be commented out because otherwise I get the error that the PhotoAsset object has no attribute called "item_type".
It works now, it is downloading photos perfectly.

I cloned that version of pyicloud and it doesn't work for me;
Traceback (most recent call last): File "/usr/local/bin/icloud", line 11, in <module> load_entry_point('pyicloud==0.9.1', 'console_scripts', 'icloud')() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/cmdline.py", line 202, in main password.strip() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 194, in __init__ self.authenticate() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/base.py", line 208, in authenticate myICloud = hack.PyiCloudService() File "/usr/local/lib/python2.7/dist-packages/pyicloud-0.9.1-py2.7.egg/pyicloud/hack.py", line 232, in __init__ super(PyiCloudService, self).__init__(self) TypeError: super() argument 1 must be type, not classobj

Hey!
You should clone my repo, that version should work. Actually, PeterHedley94's version in itself did not work for me either, that's why I had to merge his changes into ndbroadbent's pyicloud library. Both of them are necessary because ndbroadbent's repo is fully compatible with his photo downloader code and PeterHedley94's changes (adding hack.py) solves the authentication issue.
So I recommend that you pull my forks:
https://github.com/danielfmolnar/icloud_photos_downloader
https://github.com/danielfmolnar/pyicloud
Please let me know if they don't work properly.

Hi @danielfmolnar  - no luck I'm afraid.
I've pulled and installed (sudo python setup.py install) both forks but I still get the authentication error. If I run the icloud binary itself I get the repeated prompt to authenticate.

This worked for me once I install FIRST icloud_photos_downloader, and THEN pyicloud..
before, i did it in that order, i kept getting the authentication error.

This worked for me once I install FIRST icloud_photos_downloader, and THEN pyicloud..
before, i did it in that order, i kept getting the authentication error.
Edit: I am now able to authenticate and create a cookie using icloud - but icloudpd still gives me the error above ( PyiCloudAPIResponseError('Failed to validate the credentials from cookie'))
this is SURELY not the way to do it... but the way I finally made this work was to copy the base.py from the 0.9.1 installation to the 0.10.0 installation
cp /lib/python3.7/site-packages/pyicloud-0.9.1-py3.7.egg/pyicloud/base.py /lib/python3.7/site-packages/pyicloud_ipd-0.10.0-py3.7.egg/pyicloud_ipd/
I am so ashamed... someone please tell me the right way to do this...

That is indeed working for me too. I'm not to bothered about how it works so long as it does work 😄

This worked for me once I install FIRST icloud_photos_downloader, and THEN pyicloud..
before, i did it in that order, i kept getting the authentication error.
Edit: I am now able to authenticate and create a cookie using icloud - but icloudpd still gives me the error above ( PyiCloudAPIResponseError('Failed to validate the credentials from cookie'))
this is SURELY not the way to do it... but the way I finally made this work was to copy the base.py from the 0.9.1 installation to the 0.10.0 installation
cp /lib/python3.7/site-packages/pyicloud-0.9.1-py3.7.egg/pyicloud/base.py /lib/python3.7/site-packages/pyicloud_ipd-0.10.0-py3.7.egg/pyicloud_ipd/
I am so ashamed... someone please tell me the right way to do this...

Following those steps I'm getting this error message:
Traceback (most recent call last):   File "/usr/local/bin/icloudpd", line 11, in <module>     load_entry_point('icloudpd==1.4.4', 'console_scripts', 'icloudpd')()   File "/usr/local/lib/python3.7/dist-packages/click-6.7-py3.7.egg/click/core.py", line 722, in __call__     return self.main(*args, **kwargs)   File "/usr/local/lib/python3.7/dist-packages/click-6.7-py3.7.egg/click/core.py", line 697, in main     rv = self.invoke(ctx)   File "/usr/local/lib/python3.7/dist-packages/click-6.7-py3.7.egg/click/core.py", line 895, in invoke     return ctx.invoke(self.callback, **ctx.params)   File "/usr/local/lib/python3.7/dist-packages/click-6.7-py3.7.egg/click/core.py", line 535, in invoke     return callback(*args, **kwargs)   File "/home/simon/.local/lib/python3.7/site-packages/icloudpd/base.py", line 277, in main     directory = os.path.normpath(directory)   File "/usr/lib/python3.7/posixpath.py", line 340, in normpath     path = os.fspath(path) TypeError: expected str, bytes or os.PathLike object, not NoneType

Hi @danielfmolnar

So I recommend that you pull my forks:
https://github.com/danielfmolnar/icloud_photos_downloader
https://github.com/danielfmolnar/pyicloud
Please let me know if they don't work properly.

Hi @danielfmolnar - no luck I'm afraid.
I've pulled and installed (sudo python setup.py install) both forks but I still get the authentication error. If I run the icloud binary itself I get the repeated prompt to authenticate.

same for me. I did sudo pip uninstall pyicloud followed by sudo python setup.py install in your repository, and still get the following error:
icloudpd --directory tttest --username<EMAIL_ADDRESS>--recent 50
2019-06-23 19:20:11 DEBUG    Authenticating...
iCloud Password:
Traceback (most recent call last):
  File "/usr/local/bin/icloudpd", line 11, in <module>
    load_entry_point('icloudpd==1.4.4', 'console_scripts', 'icloudpd')()
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/icloudpd/base.py", line 243, in main
  File "build/bdist.linux-x86_64/egg/icloudpd/authentication.py", line 39, in authenticate
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.10.0-py2.7.egg/pyicloud_ipd/base.py", line 193, in __init__
    self.authenticate()
  File "/usr/local/lib/python2.7/dist-packages/pyicloud_ipd-0.10.0-py2.7.egg/pyicloud_ipd/base.py", line 216, in authenticate
    raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError(u'Failed to validate the credentials from cookie',))

The password was correct, I'm sure.

after pulling danielfmolnars forks and copying base.py according to gregbert42s suggestion,
i'm getting 2fa window on my iphone every time i run icloudpd. no matter if i allow or decline the request, icloudpd connects and downloads. any solution?

Same happens for me. Important to set it for awake hours :-)

On Jun 26, 2019, at 12:18 PM, biGdada<EMAIL_ADDRESS>wrote:
after pulling danielfmolnars forks and copying base.py according to gregbert42s suggestion,
i'm getting 2fa window on my iphone every time i run icloudpd. no matter if i allow or decline the request, icloudpd connects and downloads. any solution?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I believe I have solved this in a somewhat clean way.
If you download and install the versions from my two branches: (start with pyicloud-ipd), it should work.
https://github.com/andersjohansson/pyicloud/tree/pyicloud-ipd-auth-fixes
https://github.com/andersjohansson/icloud_photos_downloader/tree/fix-doc-and-version
As @ndbroadbent, says, icloud_photos_downloader uses the renamed pyicloud-ipd from the pyicloud-ipd branch (https://github.com/ndbroadbent/pyicloud/tree/pyicloud-ipd). I have merged the changes from @PeterHedleyJHA to this branch, and it all seems to work.
(I also documented the previously undocumented change that download directory has to be specified as --directory, instead as first argument, confused me for a while...)

It still seems to launch the 2FA window everytime on the phone though, no idea how to solve that.

Good work! Having the 2FA pop up all the time is not possible for me as I don't have physical access to the device. I'm hoping we can work aound that limitation

Yes, the 2FA pops up for me too.
However, I've noticed that although this code works perfectly now, it doesn't download the videos for the live photos. The issue seems to be that photo.versions no longer includes the live versions, only the images. I'm not familiar with the Apple iCloud API, but does anybody have any suggestions?

I believe I have solved this in a somewhat clean way.
If you download and install the versions from my two branches: (start with pyicloud-ipd), it should work.
https://github.com/andersjohansson/pyicloud/tree/pyicloud-ipd-auth-fixes
https://github.com/andersjohansson/icloud_photos_downloader/tree/fix-doc-and-version
(download, extract, and run python setup.py install [--user])
As @ndbroadbent, says, icloud_photos_downloader uses the renamed pyicloud-ipd from the pyicloud-ipd branch (https://github.com/ndbroadbent/pyicloud/tree/pyicloud-ipd). I have merged the changes from @PeterHedleyJHA to this branch, and it all seems to work.
(I also documented the previously undocumented change that download directory has to be specified as --directory, instead as first argument, confused me for a while...)

Hi. I am not good at python and just install icloud_photos_downloader tool by using pip command. Could I install your update with pip command too? Thanks for your help.

I try it. but reason is style same
File "c:\users\pilotmedia\appdata\local\programs\python\python37-32\lib\site-packages\pyicloud_ipd\base.py", line 193, in init
self.authenticate()
File "c:\users\pilotmedia\appdata\local\programs\python\python37-32\lib\site-packages\pyicloud_ipd\base.py", line 216, in authenticate
raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Failed to validate the credentials from cookie'))

I can't get any of the solutions above to work for me - I still get the error which ends with:
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError(u'Failed to validate the credentials from cookie',))
...no matter what I try.
I'm not convinced that I'm uninstalling properly (or even if an uninstall is necessary?)
Can anyone who has this working using any of the above solutions give me some guidance on what I need to do in order to get this working again please?

Those files helped: https://github.com/shuixin536/icloud_photos_downloader

Those files helped: https://github.com/shuixin536/icloud_photos_downloader

I try this new link. I run it on W10. reinstall python remove old script make new install with upgrade pip..
when i try run pytest
=================================== ERRORS ====================================
________________ ERROR collecting tests/test_authentication.py ________________
ImportError while importing test module 'C:\icloud\icloud_photos_downloader\tests\test_authentication.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_authentication.py:7: in
from icloudpd.authentication import authenticate, TwoStepAuthRequiredError
icloudpd\authentication.py:5: in
import pyicloud
E   ModuleNotFoundError: No module named 'pyicloud'
______________ ERROR collecting tests/test_autodelete_photos.py _______________
ImportError while importing test module 'C:\icloud\icloud_photos_downloader\tests\test_autodelete_photos.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_autodelete_photos.py:10: in
from icloudpd.base import main
icloudpd\base.py:15: in
from tzlocal import get_localzone
E   ModuleNotFoundError: No module named 'tzlocal'
Can you help me where can i resolve it?

Thanks @andersjohansson, your branches work for me.  One catch (I guess working as expected), don't include the smtp notification arguments that icloudpd provides, as including the 2fa notification settings will stop the code every time since the 2fa always triggers.  I checked my live photos and I get the videos the same as before, so I didn't share the same issue as @danielfmolnar

Hi guys, please correct me if I'm wrong. Even if the code can now authenticate and download photos, it still triggers a 2FA prompt on the iOS devices every time. There is no need to approve the 2FA request on the phone, the script can download photos without approval, but the prompt is annoying.
icloudpd.exe" --directory "C:\Temp\icloud-photo-download-test" --username<EMAIL_ADDRESS>--size original --recent 25

@markuskkkl
Same here.

@andersjohanson,
i tried your steps but it fails with:
Traceback (most recent call last):
File "/home/asus/.local/bin/icloudpd", line 11, in
load_entry_point('icloudpd==1.4.5', 'console_scripts', 'icloudpd')()
File "/home/asus/.local/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/home/asus/.local/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/asus/.local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/asus/.local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "build/bdist.linux-x86_64/egg/icloudpd/base.py", line 243, in main
File "build/bdist.linux-x86_64/egg/icloudpd/authentication.py", line 39, in authenticate
File "/home/asus/.local/lib/python2.7/site-packages/pyicloud_ipd-0.11.0-py2.7.egg/pyicloud_ipd/base.py", line 200, in init
self.setupiCloud = SetupiCloudService(self)
File "/home/asus/.local/lib/python2.7/site-packages/pyicloud_ipd-0.11.0-py2.7.egg/pyicloud_ipd/base.py", line 406, in init
super(SetupiCloudService, self).init(session)
TypeError: super() argument 1 must be type, not classobj

@andersjohansson thanks for the update, authentication works great and I can e.g. list all the album names. But when I try to download any photos it doesn't work and also doesn't give any errors:
user@host:~# docker run -it --rm --name iphoto -v /storage/iphoto:/data mdbraber/iphoto:latest icloudpd --username<EMAIL_ADDRESS>--password 'secret' --recent 10 -d /data --cookie-directory /data
2019-07-25 12:01:49 DEBUG    Authenticating...
2019-07-25 12:01:54 DEBUG    Looking up all photos and videos from album All Photos...
2019-07-25 12:01:54 INFO     Downloading 10 original photos and videos to /data/ ...
  0%|                                                                                | 0/10 [00:00<?, ?it/s]
2019-07-25 12:01:54 INFO     All photos have been downloaded!

Also when trying to use --only-print-filenames it doesn't show anything...

I believe I have solved this in a somewhat clean way.
If you download and install the versions from my two branches: (start with pyicloud-ipd), it should work.
https://github.com/andersjohansson/pyicloud/tree/pyicloud-ipd-auth-fixes
https://github.com/andersjohansson/icloud_photos_downloader/tree/fix-doc-and-version
(download, extract, and run python setup.py install [--user])
As @ndbroadbent, says, icloud_photos_downloader uses the renamed pyicloud-ipd from the pyicloud-ipd branch (https://github.com/ndbroadbent/pyicloud/tree/pyicloud-ipd). I have merged the changes from @PeterHedleyJHA to this branch, and it all seems to work.
(I also documented the previously undocumented change that download directory has to be specified as --directory, instead as first argument, confused me for a while...)

@andersjohansson, this is how I came from your post to a working solution:
git clone https://github.com/andersjohansson/pyicloud.git
cd pyicloud
git checkout pyicloud-ipd-auth-fixes
# we need to use python3, otherwise it did not work for me
python3 setup.py install --user
cd ..
git clone https://github.com/andersjohansson/icloud_photos_downloader.git
cd icloud_photos_downloader
git checkout fix-doc-and-version
python3 setup.py install --user
cd ..
mkdir temp
cd temp
icloudpd -d . --username ****** -a 'All Photos' --recent 50

I've managed it to work first by uninstalling the pip version. That's an important step.
pip uninstall icloudpd
Then I followed the steps described by @sniechzial but without appending the --user option.
I also don't need to set the 'All Photos'
So:
git clone https://github.com/andersjohansson/pyicloud.git
cd pyicloud
git checkout pyicloud-ipd-auth-fixes
# we need to use python3, otherwise it did not work for me
python3 setup.py install
cd ..
git clone https://github.com/andersjohansson/icloud_photos_downloader.git
cd icloud_photos_downloader
git checkout fix-doc-and-version
python3 setup.py install
cd ..
mkdir temp
cd temp
icloudpd -d . --username ****** --recent 50

I got it working using @SultanICQ 's commands.
I was using the docker solution and if someone is more ninja with docker I'd love to test out an updated image.

It works, thank you! @SultanICQ

Any chance you'd be willing to put that in a docker image @SultanICQ ?

Hi @hyper-ex . In fact I don't really know how to do it. I tried docker images but I really dont know how to build one with that the right way.

Hey, everyone.
I'm no docker ninja, but I built my own container based on Alpine Linux and should be able to incorporate these fixes. I'll look at it Tue/Thus night.

That’d be absolutely awesome!

3 sep. 2019 kl. 02:12 skrev boredazfcuk<EMAIL_ADDRESS>
Hey, everyone.
I'm no docker ninja, but I built my own container based on Alpine Linux and should be able to incorporate these fixes. I'll look at it Tue/Thus night.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi everyone! It would be great if someone was able to submit a PR with these changes, where all the tests are passing, and with 100% test coverage (and no problem if you need to delete some code to remove some features. We can re-add something later if there is enough demand.) Cheers!

I think the problem with the current workaround is that it’s a bit of a hack, and it breaks some functionality that other users are depending on. But either way, it would be better to have a working script than a broken one!

Fully agree on that point @ndbroadbent =)

I think everyone might be on a wild goose chase...
I have this working in a docker container and I haven't applied any fixes at all. The only issue is this error when generating the cookie:
Traceback (most recent call last):   File "/usr/bin/icloudpd", line 11, in <module>     load_entry_point('icloudpd==1.4.4', 'console_scripts', 'icloudpd')()   File "/usr/lib/python3.7/site-packages/click-6.7-py3.7.egg/click/core.py", line 722, in __call__     return self.main(*args, **kwargs)   File "/usr/lib/python3.7/site-packages/click-6.7-py3.7.egg/click/core.py", line 697, in main     rv = self.invoke(ctx)   File "/usr/lib/python3.7/site-packages/click-6.7-py3.7.egg/click/core.py", line 895, in invoke     return ctx.invoke(self.callback, **ctx.params)   File "/usr/lib/python3.7/site-packages/click-6.7-py3.7.egg/click/core.py", line 535, in invoke     return callback(*args, **kwargs)   File "/usr/lib/python3.7/site-packages/icloudpd-1.4.4-py3.7.egg/icloudpd/base.py", line 277, in main   File "/usr/lib/python3.7/posixpath.py", line 340, in normpath     path = os.fspath(path) TypeError: expected str, bytes or os.PathLike object, not NoneType
But I only get the 2FA prompt on my iDevice when I generate my cookie. Every sync after that works as expected.

I’m gonna have to double check but I’m pretty sure that’s not what I’m getting.

what is the status of 2FA prompt right now? has anyone succeeded to to make it work without triggering the 2FA dialog?

I’m gonna have to double check but I’m pretty sure that’s not what I’m getting.
Is this the same version that’s in @ndbroadbent s container?

No, it's using a container I built... Sorry, I should have linked it: https://github.com/boredazfcuk/docker-icloud_photos_downloader

what is the status of 2FA prompt right now? has anyone succeeded to to make it work without triggering the 2FA dialog?

Yes: https://github.com/boredazfcuk/docker-icloud_photos_downloader

Sorry, about line 24. I've not had chance to break it out into it's own file yet as I write most of my stuff in Portainer's web editor and I can't feed additional files into it...

what is

what is the status of 2FA prompt right now? has anyone succeeded to to make it work without triggering the 2FA dialog?

Yes: https://github.com/boredazfcuk/docker-icloud_photos_downloader
what is the right set of commands to install it directly on linux/freebsd (without the docker)?

what is the status of 2FA prompt right now? has anyone succeeded to to make it work without triggering the 2FA dialog?

Yes: https://github.com/boredazfcuk/docker-icloud_photos_downloader

what is the right set of commands to install it directly on linux/freebsd (without the docker)?

The dockerfile is just an install script which has all the commands needed to install it on Alpine Linux.
You should be able to adapt it to install on whatever distribution you like.

I've finished tidying the code. I've split out the launch script to a separate file now so it's easier to read.
I've re-built the Docker image and the container from scratch three or four times now and it definitely doesn't have the 2FA prompting issue.
This issue must be some sort of system issue, rather than code problem. If people would verify my findings and report any issues that would be great, thanks.

Volumes:
The --volume statement in your example is hardcoded to boredazfcuk but the script actually checks for /home/${USER}/iCloud. Suggestion would be to use something generic such as /data or /photos instead.
Script init:
During script init you check for /home/${USER}/iCloud/.mounted but that file is never created (unless you do it manually)
CLIOPTIONS:
It seems the script only allows passing of one set of parameters, when I tried to pass both smtp-username and smtp-password it failed.
Finally, and sadly:
Even after fixing those things, it does not sync any photos at all, my photos directory is empty (except for .mounted). See output below:
/ # /usr/local/bin/sync-icloud.sh
2019-09-04 10:44:58 INFO     Local account: hyper-1024:sc-media-65537
2019-09-04 10:44:58 INFO     Apple account: zzzz@yyyy
2019-09-04 10:44:58 INFO     Cookie directory: /config
2019-09-04 10:44:58 INFO     iCloud directory: /home/hyper/iCloud
2019-09-04 10:44:58 INFO     Command line options:
2019-09-04 10:44:58 INFO     Syncronisation for hyper started
2019-09-04 10:44:58 INFO     Sync hyper iCloud...
Usage: icloudpd
Error: Got unexpected extra argument ()
/usr/local/bin/sync-icloud.sh: line 79: 2019-09-04 10:45:03 INFO     Exit code non-zero - Error: 0: not found
2019-09-04 10:45:03 INFO     Set correct file time stamp from exif data...
2019-09-04 10:45:03 INFO     Set group and permissions of downloaded files...
2019-09-04 10:45:03 INFO     Syncronisation for hyper complete
2019-09-04 10:45:03 INFO     Next syncronisation at 16:45

Hi,
Thanks for testing. Thanks for pointing the bits you're having issue with... hopefully some answers:
Volumes point 1:
The --volume statement in the readme has hardcoded values because it's only an example. It's the exact command I used to create my working container. I will expand the readme more to show that.
Volumes point 2: I use /home/${USER}/iCloud as the download destination within the container as I have 3 different accounts that I sync (me, Mrs & kids). Having them sync to the correct user's home folder means I have an easy visual guide as to which container I'm operating in if I am connected via the shell. They also map to the same location on the host machine for consistency. I could change the download path within the container to something generic, but I see that as a drawback as I'd lose user visibility. Also, it doesn't really matter where they're stored within the container as the destination folder is a bind mount to a different location on the host.
Script init:
Sorry, I should have made it clear that the .mounted file needs to be created manually. I can't automate it as the container has no way of knowing if the underlying volume is mounted or not. I've updated the readme to reflect this.
CLIOPTIONS:
What options did you use? I'm guessing that it's probably a string expansion issue as I tested multiple options with
 --env CLIOPTIONS="--folder-structure={:%Y} -- recent 50"

Hey,
Ah, got it, thanks.
After running the script inside the container I get the following
/usr/local/bin # /usr/bin/icloudpd --directory \"/home/${USER}/iCloud\" --cookie-directory=\"${CONFIGDIR}\" --username
=\"${APPLEID}\" --password=\"${APPLEPASSWORD}\" \""${CLIOPTIONS}"\""
> "
Usage: icloudpd <options>

Error: Invalid value for "-d" / "--directory": Path ""/home/hyper/iCloud"" does not exist.

Could there be an issue with the issues with the escaping?

This error is saying that the destination folder for the downloads does not exist inside the container.  Are you sure your container was created with two volumes and not just one?
--volume icloudpd_boredazfcuk_config:/config \
--volume /home/boredazfcuk/iCloud:/home/boredazfcuk/iCloud \

The first volume is for storing the authentication token so it's available between restarts.
The second volume is basically ":"
The "/home/hyper/iCloud" folder should automatically be created when you specify that second volume.
I've updated the readme so it should be clearer now.

Hey,
Yupp, /home/hyper/iCloud most definitely exists:
/home/hyper/iCloud # ls -al
total 0
drwxrwx--- 1 hyper sc-media 28 Sep  4 10:41 .
drwxr-sr-x 1 hyper sc-media 12 Sep  4 10:42 ..
-rw-rw---- 1 hyper sc-media  0 Sep  4 10:41 .mounted
drwxrwx--- 1 hyper sc-media  8 Aug 28 08:45 @eaDir
/home/hyper/iCloud #

Path ""/home/hyper/iCloud"" does not exist.

I think it's something odd is happening with the quotes. I think those double quotes means it's actually using "/home/hyper/iCloud" (quote included) as the path

I suppose I can really remove those, as usernames with spaces in them are not allowed. I'm just in the habit of quoting all my variables so I can deal with spaces.

I also noticed that sometimes you use mix space and = when assigning the parameters

I copy/pasted a lot of the script from one I created to run in an LXC container. That was when the instructions included '='. They've since been removed. See here: https://github.com/ndbroadbent/icloud_photos_downloader/issues/119

If I remove all escaped quotes (") the command runs like a charm..

Just to confirm, you don't get the 2FA prompting issue?

No issues with the 2FA!

Winner! Winner!

Thanks @boredazfcuk and @hyper-ex for building this. Downloading photos now works great (without 2FA prompt) for all "album-Albums" (i.e. named albums I've created my self). For the "official" Apple albums like All Photos, Slo-mo. Screenshots or others it doesn't work ("0 photos"). Any idea what could be going on or how to best debug? Thanks!

@mdbraber Sorry for the late reply, I was busy building a bunch of other containers and forgot to respond...
I've never used the --album functionality to download named albums. I just run it without that option and it pulls "All Photos" which includes all the screen shots, videos etc.
Also, I've now pushed this to Ducker hub so the image can be downloaded with:
docker pull boredazfcuk/icloudpd

I am super appreciative of this... So happy to have it working again
seamlessly.
On Mon, Sep 16, 2019 at 1:06 PM boredazfcuk<EMAIL_ADDRESS>wrote:

@mdbraber https://github.com/mdbraber Sorry for the late reply, I was
busy building a bunch of other containers and forgot to respond...
I've never used the --album functionality to download named albums. I just
run it without that option and it pulls "All Photos" which includes all the
screen shots, videos etc.
Also, I've now pushed this to Ducker hub so the image can be downloaded
with:
docker pull boredazfcuk/icloudpd
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ndbroadbent/icloud_photos_downloader/issues/139?email_source=notifications&email_token=AEBNBYMQIS4SKNJL7WXTBZLQJ5SDZA5CNFSM4HXETO72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6YZOCY#issuecomment-531732235,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEBNBYOROYJ5Y2AGXDN5XK3QJ5SDZANCNFSM4HXETO7Q
.

Very happy! Great job and thanks @boredazfcuk  @hyper-ex !
Compared to my previous build I'm now missing the --album parameter. Please correct me, hope I'm just wrong and the feature to download specific albums is still available.

This issue is resolved, I guess, @ndbroadbent

Thanks @menkej, will close!


Auto Copy on Select

I would like to be able to select "some text" then on the release of the left mouse button trigger command+c. I can get the Left click to copy, but when I do I am no longer able to select any text. No sure if it's possible, but I figured this would be the best place to ask. Thanks!

Karabiner can not detect whether you selected text or not.
Therefore, Karabiner always sends command+c for each releasing of the left button to achieve auto copy. (You can do it by the following autogen.)
But this setting forbits double click.
You should consider to use another trigger.
    <autogen>
      __KeyToKey__
      PointingButton::LEFT,
      PointingButton::LEFT,

      Option::KEYTOKEY_AFTER_KEYUP,
      KeyCode::C, ModifierFlag::COMMAND_L,
    </autogen>

Would it be possible to, after dragging while still holding the left click to click the space bar and have that perform the copy function? Would a long click to copy or a delay before copying help? Again I totally appreciate this. It is as close as I've come to getting that functionality.
For anyone who comes across this looking for the same thing. Here's a little background.
I tried to create an automator service to copy selected, but it doesn't work in every app. Then I tried to edit your code a bit to run shell call to command+c in applescript, still can't double click. Then I started using your  tag to filter out apps. Maybe it is not meant to be, haha! Maybe someday Ill learn objc, pop clip is so close too, but you can't programmatically check to see if pop clip bubble is up, but you can control the bubble with applescript.

Would it be possible to, after dragging while still holding the left click to click the space bar and have that perform the copy function?

I think you should work to get the selection text instead of tuning the command+c timing.
You might be able to get the selection text by Accessibility APIs.
http://stackoverflow.com/questions/6544311/how-to-get-global-screen-coordinates-of-currently-selected-text-via-accessibilit
Then, make shell command which sends command+c if the selection text exists.
Call the command from Karabiner from the above autogen, you can copy text in suitable situation even if you call the command for each button release.
https://pqrs.org/osx/karabiner/xml.html.en#vkopenurldef


API 0.7: Adjust supported API version

We'll need to adjust the API version here from 0.6 to 0.7:
https://github.com/Malax/libcnb.rs/blob/7a8092a6d879c48ee0ee623002b77ad9b69bcf66/libcnb/src/lib.rs#L40-L41
We'll also likely want to bump the API version used in various test fixtures, for completeness.
GUS-W-10426513.

Duplicate of #271 (bumping straight from 0.6 to 0.8).


updated delete person in User Guide

Coverage remained the same at 92.174% when pulling d509c0ff0d6fffafd5aee1ef54a48c0b14471453 on aziziazfar:v1.1_update_UG_DG into 6cd5adc0931f164d002f7519c68da91dc11420a3 on CS2103AUG2017-T15-B4:master.

Coverage remained the same at 92.413% when pulling 82289252f2a5f5545c518ffa20f8960a386da215 on aziziazfar:v1.1_update_UG_DG into 31004efe7a9e5785fbdb93d4c4fe2d5c7d137047 on CS2103AUG2017-T15-B4:master.


Picasso options in SliderLayout

Hi,
I'm using your library to display sliding images and I'm wondering if it would be possible to add custom Picasso properties to the sliding images. I'm especially interested in adding the onlyScaleDown method to the picasso loader so big images don't get resized up. Would that be possible?
Thanks

I've sent a pull request with these changes
https://github.com/daimajia/AndroidImageSlider/pull/194


[FLINK-21223][python] Support to specify the output types of Python UDFs via string

What is the purpose of the change
This pull request will support to specify the output types of Python UDFs via string
Brief change log

Support to specify the output types of Python UDFs via string

Verifying this change
This change added tests and can be verified as follows:

test_all_data_types_string and orginal tests in test_udtf and test_udaf

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@flinkbot run azure


[FR] - Drop dingo?

@kslimani What do you think about dropping dingo for the following reasons:

Reduce 3rd party dependencies
Encourage API Evolution instead of API Versioning (https://blog.apisyouwonthate.com/api-evolution-for-rest-http-apis-b4296519e564)
Change Fractal in favor of native Resources (https://laravel.com/docs/5.6/eloquent-resources)

Also for the Spec change API Blueprint in favor of OpenAPI 3.0 spec.

@joselfonseca "API evolution" is the methodology i already use for my applications.
By dropping Dingo, you lose the "mime based" ressource versioning feature, but you can still use "URL based" ressource versioning. (so I am not against dropping Dingo dependency).
About replacing Fractal by Eloquent Resources : i have not yet myself tested this feature. It seems powerfull but i don't know how it handle the pagination for collection, especially the links in meta.
Do you know if it is automatically handled by Laravel or do you have to generate the url links yourself  ?
(if not, it may be required to create some helpers to reproduce the Dingo collection metadata)
About the OpenAPI spec i don't know, i think it may depends on the available tools to generate api documentation. (Aglio is pretty cool for API blueprint format).

I have read the pagination part for API resources, i was just wondering how the urls in "links" and "meta" are generated in the provided example :
// Copy paste from https://laravel.com/docs/5.6/eloquent-resources#pagination
use App\User;
use App\Http\Resources\UserCollection;

Route::get('/users', function () {
    return new UserCollection(User::paginate());
});

Yea, I was looking into that and got this:
[
  "data" => array:0 []
  "links" => array:4 [
    "first" => "http://localhost/api/users?page=1"
    "last" => "http://localhost/api/users?page=2"
    "prev" => null
    "next" => "http://localhost/api/users?page=2"
  ]
  "meta" => array:7 [
    "current_page" => 1
    "from" => 1
    "last_page" => 2
    "path" => "http://localhost/api/users"
    "per_page" => 20
    "to" => 20
    "total" => 31
  ]

Using
    /**
     * Returns the Users resource with the roles relation.
     *
     * @param Request $request
     * @return mixed
     */
    public function index(Request $request)
    {
        $paginator = $this->model->with('roles.permissions')->paginate($request->get('limit', config('app.pagination_limit')));
        if ($request->has('limit')) {
            $paginator->appends('limit', $request->get('limit'));
        }

        return UserResource::collection($paginator);
    }

data is empty because I removed it, but the user data is there. I am only worried about like the includes, have not seen how that works.

What happen if instead of /users you rename to something else ?
Route::get('/foobar', function () {
    return new UserCollection(User::paginate());
});

Does it properly generate links like http://localhost/api/foobar?page=1 ?

It actually does XD
  "links" => array:4 [
    "first" => "http://localhost/api/foo?page=1"
    "last" => "http://localhost/api/foo?page=2"
    "prev" => null
    "next" => "http://localhost/api/foo?page=2"
  ]
  "meta" => array:7 [
    "current_page" => 1
    "from" => 1
    "last_page" => 2
    "path" => "http://localhost/api/foo"
    "per_page" => 20
    "to" => 20
    "total" => 31
  ]

Route::group(['prefix' => 'foo'], function () {
        Route::get('/', 'Users\UsersController@index');

/**
     * Returns the Users resource with the roles relation.
     *
     * @param Request $request
     * @return mixed
     */
    public function index(Request $request)
    {
        $paginator = $this->model->with('roles.permissions')->paginate($request->get('limit', config('app.pagination_limit')));
        if ($request->has('limit')) {
            $paginator->appends('limit', $request->get('limit'));
        }

        return UserResource::collection($paginator);
    }

That's pretty cool !

Lets leave this for later XD.


BUG:Remove rows without sample name & improve docs

Rows with empty sample names are now removed from the dataframe, note that
these rows could include relevant data in other columns but since we have no
sample name, they are useless.
The documentation for read_csv was lacking a bit, so I expanded on the reason
why the arguments are being passed the way they are.
Fixes #746

:+1:

Coverage increased (+0.0%) when pulling 34b6c39208d8c9714b944cc6930138bc61a23924 on ElDeveloper:issue-746 into 91d5ffab26b2c173c0e2a08fb615592d53b78b32 on biocore:master.

Coverage increased (+0.0%) when pulling 34b6c39208d8c9714b944cc6930138bc61a23924 on ElDeveloper:issue-746 into 91d5ffab26b2c173c0e2a08fb615592d53b78b32 on biocore:master.


feat(selectionBar): Add a showSelectionBarEnd option

Add a showSelectionBarEnd option to display the selection bar after the value.
As requested in #213
Demo: http://jsfiddle.net/ValentinH/4z70y6kx/

Current coverage is 99.67%

Merging #214 into master will decrease coverage by -0.33% as of a206aff

@@            master    #214   diff @@
======================================
  Files            1       1
  Stmts          615     622     +7
  Branches         0       0
  Methods          0       0
======================================
+ Hit            615     620     +5
  Partial          0       0
- Missed           0       2     +2

Review entire Coverage Diff as of a206aff

Powered by Codecov. Updated on successful CI builds.


Use MIME::Types.of instead of hacked-on intrusive monkey patch

The monkey patch sidestepped MIME::Types accessors and broke when using RUBY_MIME_TYPES_LAZY_LOAD.

Thanks for your patch! The other patch is functionally identical, just uses type_for() rather than the of() alias.
I'm actually going to take a slightly different approach to maintain the old behavior when something like foo.xml is passed. (The monkey patch will still render it as foo.xml, whereas calling MIME::Types.type_for will render it as application/xml.)
Fixed in https://github.com/rest-client/rest-client/issues/354

Thanks!


Remark directive is not handling block contents

Initial checklist

[X] I read the support docs
[X] I read the contributing guide
[X] I agree to follow the code of conduct
[X] I searched issues and couldn’t find anything (or linked relevant results below)

Affected packages and versions
2.0.1
Link to runnable example
No response
Steps to reproduce
The remark-directive does not handling inline block contents.

Expected behavior
its should return the expected result as like remark-shortcodes.
Actual behavior
It not handling this scenario which i have mentioned above.
Runtime
Node v17
Package manager
npm 8
OS
macOS
Build and bundle tools
No response

I cannot look into your computer or your head. I have no clue what you are asking.
I think you’re saying that remark-directive has a different syntax than remark-shortcodes.
That is correct. And intentional. This project is not that project.
See the readme: https://github.com/remarkjs/remark-directive


[xDS Proto] Pin PyPI dependencies in Dockerfiles

Spin off PR of https://github.com/grpc/grpc/pull/25272

Known issue: https://github.com/grpc/grpc/issues/21858


[3.x] Fix multiple issues with CSGPolygon

3.x version of #49313.
Note: Although I've marked this as breaks compat, because it changes the way the shapes are created by fixing the way MODE_PATH ROTATION_PATH modes affect extrusion rotations, changes the UV material mappings, reverses the way MODE_PATH normals are defined (i.e. no longer requires them to be inverted see comment), and restricts the valid values for MODE_PATH's path_interval, I still think the quality of the fixes make it viable for 3.4, but the changes would need to be mentioned in the release notes.

For my game, I need to have path set to 0.25 for csg mesh to work, but in this case I guess it wouldn't work.

Updated to fix bug identified by @stebulba here, and increased the granularity of path_interval in the editor to 0.05 to allow values like 0.25.

Updated with suggested changes to the documentation included in #49313.

Thanks!


Error: get key failed from google in version 0.0.3

Hello zlargon,
Currently, I'm using google-tts-api in version 0.0.3, and it's no longer working, an "get key failed from google" error appears.
Does it not work in older versions?
Do we need to update to the latest version?
Thank.

In node_modules/google-home-notifier/package.json,
change
"google-tts-api": "0.0.3", to "google-tts-api": "0.0.4",
then update npm
cd node_modules/google-home-notifier npm update google-tts-api
works fine for me.

@chohaku84 Thank you for the reply. 👍


TextFileInput changes

Alex, what do you think about changing the calls to TextFileInputUtils.getLine() in other steps such as S3 and CSV over to use OldTextFileInput.getLine() instead?  The code is nearly identical, but I think it might be safer to keep the other steps referencing the old code until they have specifically been refactored themselves.

calls to TextFileInputUtils.getLine()
You are right. Will change.

Hmm.. in that case, shouldn't you change all the other references to OldTextFileInput in those steps to the new ones as well?  Otherwise, that pull request would involve a bunch of changes in addition to the deletes.

Can you look at creating pull requests for the other two projects that failed when we tried to merge this in Friday as well?


refactor(colors): replace global variables with setters

Codecov Report

Merging #7 (e7cf218) into master (86fae9a) will increase coverage by 0.07%.
The diff coverage is 46.66%.

@@            Coverage Diff             @@
##           master       #7      +/-   ##
==========================================
+ Coverage   41.84%   41.91%   +0.07%
==========================================
  Files          19       19
  Lines        3427     3433       +6
==========================================
+ Hits         1434     1439       +5
- Misses       1921     1922       +1
  Partials       72       72

Flag
Coverage Δ

go1.13
41.91% <46.66%> (+0.07%)
:arrow_up:

go1.14
?

go1.15
?

go1.16
?

go1.17
41.91% <46.66%> (+0.07%)
:arrow_up:

macos-latest
?

ubuntu-latest
41.91% <46.66%> (+0.07%)
:arrow_up:

unittests
41.91% <46.66%> (+0.07%)
:arrow_up:

windows-latest
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

colors/colors.go
42.30% <46.66%> (+2.51%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 86fae9a...e7cf218. Read the comment docs.


Linting and PEP8 compliance

I have been away from the codebase for a while and ouch, my eyes. I have used pyflakes or flake8 assiduously for Python development for a long time now and it seems that conventional style isn't being followed in the codebase:
$ flake8 pandas/core | wc -l
2047
It's a big codebase so adding a pre-commit hook might be too onerous, but can we get flake8 back to zero (with some well-placed # noqa marks on unintentionally-unused-imports)?
Secondly, perhaps we should add a Readability or Style label for these kinds of issues?

See also https://github.com/pydata/pandas/issues/8317 and https://github.com/pydata/pandas/issues/6248
So there has been some discussion before, and in principle we ask of PRs to be PEP8 compatible, only we do not really enforce it in practice.
To better enforce it on new code, something like checking for this in the test suite (eg matplotlib does something like this) is probably a better idea than manual reviewing.

Okay. IMHO the code smell has reached pretty intense levels. I suggest we engage in a cleanup to get flake8 passing cleanly. I see concerns in #6248 that it will break PRs and git blame, and this is true, but it's the price we pay for having not been more fastidious on an on going basis. Let's please stick to 80 characters per line.

So prob a lot my fault. I tend to use long lines. :)
[jreback-~/pandas] flake8 pandas/core/|wc
    1922   15211  134394
[jreback-~/pandas] flake8 pandas/core/ | grep 'line too long'|wc
     924    8316   66073
[jreback-~/pandas] flake8 pandas/core/ | grep 'missing whitespace' | wc
     351    2106   23052
[jreback-~/pandas] flake8 pandas/core/ | grep 'never used' | wc
      24     264    2135
[jreback-~/pandas] flake8 pandas/core/ | grep 'expected 2 ' | wc
     131    1048    8789

How about hooking a check into GitHub? Either a service such as Landscape Code Climate. Many of these have a broader scope but could be restricted to checking style only.
If that were enforced for new commits, the library would get there over time...

I would be strongly +1 on adding a flake8 check to our Travis CI builds to make linting errors be failed buidls. Any thoughts on how to proceed with the cleanup?

I have not been impressed by the reliability of Landscape as a CI system.
That said, I'm +1 for giving this a try and taking a more stringent approach going forward. This should give me a little more leverage when reviewing @jreback's long lines with missing whitespace :).

hah, this turned out to be easy to fix in for example this PR: https://github.com/pydata/pandas/pull/11892

Perhaps to get our training wheels let's fix the style in pandas/core and add a flake8 pandas/core to Travis and see how it goes?

okie dokie (always wanted to write that)!

I'm not sure what's the best practice for using flake8 in CI; I've used prospector but it combines pylint with pep8 which is a little too draconian (pylint gets grumpy about a host of things).

prospector supports pyflakes and pep8, here's an example config that runs pep8 only, probably not hard to add pyflakes to it
https://github.com/cloudera/ibis/blob/master/.landscape.yaml

I would strongly support the way you're about to go and will do follow the rule for my further contribution. but I have a tiny concern on it. Fixing the style in pandas/core would generates huge diffs in one commit and make it difficult to take a look into changes in commit history. I just want to know that how you guys thinking about this?

#11941 will now report the linting as a separate section on the travis builds, won't fail them yet. But as soon as we are done conversions (and after pandas/core, can do other sections), can simply flip the switch.
I am now going to be doing a check on PR's and promoting: git diff master | flake --diff.

It was one thing to have code that follows PEP8 (and will pass the flake8 CI test), but this does not always say something about the code style itself.
Eg the discussion about using parentheses instead of \ (although this is actually quite explicit in PEP8), see https://github.com/pydata/pandas/pull/11945#commitcomment-15246506 and https://github.com/pydata/pandas/pull/11951#issuecomment-168562469. Or other possible discussion about how to do hanging idents, ..
What are the opinions about using some tool for that as well? Eg https://github.com/google/yapf

Might be worth exploring a tool if it will do what we need. I know I have some style preferences that are inconsistent with PEP8.

The motivation from https://github.com/google/yapf:

The idea is also similar to the 'gofmt' tool for the Go programming language: end all holy wars about formatting - if the whole code base of a project is simply piped through YAPF whenever modifications are made, the style remains consistent throughout the project and there's no point arguing about style in every code review.

As an example of the output of YAPF, I quickly ran it on the same files as @rockg adapted manually or with autopep8 in https://github.com/pydata/pandas/pull/11951 =>  https://github.com/pydata/pandas/pull/11956

How are we doing here? Can I help?

I'm personally waiting for some consensus.  I vote for using YAPF with some manual modifications.  In particular the error strings @wesm mentioned (was the example in the PR pandas preferred style?) along with some weird parentheses splitting that I find weird.  If this is agreeable, then I can do the other files this way.  I did not like the output of autopep8 versus what @jorisvandenbossche was able to produce via YAPF.

@rockg ok, why don't you reformat the PR #11951 with YAPF / manual changes as need, and we'll close #11956 and can proceed?

+1

Let me know when I can proceed with more code reorg and test refactoring. It would be nice if we could clean up pandas/tests since I fixed all the style issues manually (with a touch of autopep8 for the each stuff) in the test_frame break up.

I think ok to proceed as you see fit. other than collisions with @rockg general cleanup & a couple of larger PR's (the resample one, but that only touches a couple of files), and @shoyer IntervalIndex. its not a big deal to rebase.

OK, I will start working on pandas/tests and pandas/tseries/tests

I'm going to take a crack at pandas/tseries. It would be really great if we can get flake8 warning == build failure by end of the month.

The ugliest subpackage left is pandas/io. Takers? I can take pandas/tools

One thing that we need to think about is how to prevent yaps from removing certain style choices.  For example, some formatting choices yapf makes are not very nice looking and I have manually fixed them.  However, if yapf is subsequently run on these files, that bad formatting will be returned. There is a way to disable yapf from formatting sections of code, but I'm not too happy about having to go back and add this.  Perhaps this can be considered a one-time process and understood flake8 will catch subsequent issues?  In order to make yapf work the way we want, changes will need to be made to it.

I have to say: I really don't like yapf from my trying to use it and have ended up doing most of the work manually in Emacs. I don't think we should use it after we are done here on this first round.

I usually yapf it and then go through the file manually and then do a final check with a diff.  I agree with your suggestion of not using it for subsequent style cleanups.

we're at the home stretch
08:34 ~/code/pandas  (master)$ flake8 pandas | wc -l
986
08:37 ~/code/pandas  (master)$ flake8 pandas/stats/ | wc -l
170
08:38 ~/code/pandas  (master)$ flake8 pandas/computation/ | wc -l
82
08:39 ~/code/pandas  (master)$ flake8 pandas/core/ | wc -l
196
08:39 ~/code/pandas  (master)$ flake8 pandas/sparse/ | wc -l
191

I'll do pandas/sparse.

OK, we are down to 285 warnings overall. 196 of these are pandas/core/groupby.py. As soon as #11841 is merged let's get to zero and then make flake8 warnings fail the build?

sounds good

ok, linter is now enabled on Travis-CI!
it will run tests, then check for any failures, which will fail the build.

Woo, thanks @jreback @rockg and all who worked on this for the huge effort. Clean code makes productive teams! 😊


Swashbuckle and triple-slash comments

I'm sure it's something I'm doing wrong, but I can't find documentation on what it is.  According to this page, "adding triple-slash comments to an action enhances the Swagger UI by adding the description to the section header."  That's not happening, and there's so little information about it here that I don't even know where to begin to debug it.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 14fce65f-1337-58cb-2193-8d8e0e62792d
Version Independent ID: 59222460-a971-e76e-4fe9-23d4d7d768ee
Content: Get started with Swashbuckle and ASP.NET Core
Content Source: aspnetcore/tutorials/getting-started-with-swashbuckle.md
Product: aspnet-core
Technology: aspnetcore-tutorials
GitHub Login: @zuckerthoben
Microsoft Alias: scaddie

Hi @plettb. I just tried it out myself and it is working.

Maybe you missed one of the following steps?
Modify .csproj
<PropertyGroup>
  <GenerateDocumentationFile>true</GenerateDocumentationFile>
  <NoWarn>$(NoWarn);1591</NoWarn>
</PropertyGroup>

Configure AddSwaggerGen  to enable XML comments
services.AddSwaggerGen(c =>
{
    c.SwaggerDoc("v1", new OpenApiInfo { Title = "WebApplication1", Version = "v1" });
    var xmlFile = $"{Assembly.GetExecutingAssembly().GetName().Name}.xml";
    var xmlPath = Path.Combine(AppContext.BaseDirectory, xmlFile);
    c.IncludeXmlComments(xmlPath);
});