FirebaseCoreSwift additive API

This is an additive, internal only Swift SDK that lets other products
access internal FirebaseCore APIs in a reflective but easy to consume
manner.
TODO:

[ ] Evaluate the best place  to call registerLibrary: from our extensions*
[ ] Add component container and migrate MLModelDownloader to use it
[ ] Add CocoaPods library if needed
[ ] Discuss if it's a reasonable implementation for #8973

none of our extensions have a base class or anything that gets called without user interaction (unlike +initialize). There's not a great spot to call registerLibrary: as far as I can see.

none of our extensions have a base class or anything that gets called without user interaction (unlike +initialize). There's not a great spot to call registerLibrary: as far as I can see.

One (not ideal) thought is to define a class visible to Obj-C and then use NSClassFromString inside the standard Core and check if the class exists:
@objc(FIRCoreSwift)
private class FirebaseCoreSwift: NSObject {}

This is similar to our existing but not-continued CoreDiagnostics code: https://github.com/firebase/firebase-ios-sdk/blob/bdac2b87f6d6e604deae914a970a75136b3a62b5/Firebase/CoreDiagnostics/FIRCDLibrary/FIRCoreDiagnostics.m#L378-L381. Definitely fragile and requires Core to know about all the Swift extensions, but it should work.

Thanks for the review! These changes aren't needed right now and it'll be pretty easy to bring them in after the fact so I'm going to close this as a proof of concept and can re-open or copy-paste as needed from here.


List of TODOs

Add videos to main page
Emergency tab in Help section
Jobs & Skills section
Fix icon on win32 version
Horizontal scroll on macos
Fix reading cache file so it doesn't throw exceptions
Fix website icon
Validation to admin board

http://pgicons.abiro.com/

For emergency section add table for contact information and some text saying calm down, for jobs & skills lots of text with an image to the right possibly a link


The “moe” address for starting multiple colabs is the same

The other three are all different addresses, and only the moe is the same

Op,It seems to be in the wrong place.


setData stopped working in my flutter application.

I've been using firebase plugins inside my flutter app successfully for reading and writing data to firestore.
However from last month, I'm not able to execute
await Firestore.instance
.document('temp/w')
.setData({'hello': 1}).whenComplete(() {
print('done!');
});
But, get()  is working fine.
await Firestore.instance
.document('users/$username')
.get()
.then((DocumentSnapshot doc) {
print('123: ${doc['username']}');
});
I found
https://github.com/firebase/firebase-ios-sdk/issues/2167#issuecomment-445929541
this but I have already signed in and authorized  the user.

I figured out that my app was exceeding the daily 50k free writes of firebase spark plan.
But I didn't get any error in my log, mail or notification on my console. There should be some way to get notified.
I hope this will help others having similar issue.


Exclude getters and setters from mapping

If I could, I would transform this issue to "Ignore annotation not working on getters".
It seems that the ignore annotation is not applied on entity getters, and the generator doesn't creates any errors.
I did not tried on setters.


Option to disable CSS validation

Hi. IDE throws warnings "semi-colon expected" while using Tailwind (even with official Tailwind extension enabled).
example.scss
.class {
  @apply inline-block font-bold text-center py-3 px-10 transition-colors duration-200;
}

This is common issue, and to fix that we need to disable build-in CSS/SCSS validation and install Stylelint extension to use own config. Then in stylelint.config.js config rule to suppress wrong warning:
"declaration-block-trailing-semicolon": null,
Simple, it works.
But this does not work for components. For some reason Volar is ignoring vscode style validation disabled option. It still throws

semi-colon expected  scss(css-semicolonexpected)

I found that in Vetur extension there is separate option to disable build-in css validation. And indeed there it works.
So my feature request - same option in Volar or respecting setting from IDE :)

Ok, this is duplicate of issue #69


RSC Typeings for getStoryblokApi() broken, all methods have any as type

Describe the issue you're facing
The type of const storyblokApi = getStoryblokApi(); is correctly StoryblokClient, but the resolution of the methods is broken.
All methods have any as type, not very usefull. I followed another issue saying for RSC types to work you have to change out moduleResolution in tsconfig.json to node16. After that the rsc import gets picked up by tsserver, but still no methods.

Does someone else also experience this?
Reproduction
https://github.com/storyblok/storyblok-react/tree/main/playground-next13-rsc
Steps to reproduce
No response
System Info
System:
    OS: macOS 13.3.1
    CPU: (10) arm64 Apple M1 Pro
    Memory: 639.28 MB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 16.13.1 - ~/.nvm/versions/node/v16.13.1/bin/node
    Yarn: 1.22.17 - ~/.nvm/versions/node/v16.13.1/bin/yarn
    npm: 8.1.2 - ~/.nvm/versions/node/v16.13.1/bin/npm
    Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 114.0.5735.133
    Safari: 16.4

Used Package Manager
yarn
Error logs (Optional)
No response
Validations

[X] Follow our Code of Conduct

Hi @carlonoelle thanks for your report.
I'm trying to replicate the issue using the playground-next13-rsc from the SDK repository, and it looks like working correctly for me.
Do you have any other detail or step that can help us to reproduce the issue?

I am having the same issue.
Steps to reproduce:

Create a new next app using create-next-app
Add import statement ➡ import { storyblokInit, apiPlugin } from "@storyblok/react/rsc";

@carlonoelle can you check if everything is ok? Or you have any way to share the scenario is not working for you?

I have a similar problem, but in my case the "rsc" folder is not typed at all:

If I change my tsconfig to use moduleResolution: "NodeNext", I can import from /rsc, but then I get the same problem as OP:

I simply created a new NextJs project using "npx create-next-app@latest" and then installed @storyblok/react using "npm install @storyblok/react".

Hi @addemod thanks for sharing details.
The method getStory is part of StoryblokJSClient:
https://github.com/storyblok/storyblok-js-client/blob/7dfdd54961add726c392738d6c962a08ed3d6dfd/src/index.ts#L271
That client is imported and later exported by the Storyblok React SDK:
https://github.com/storyblok/storyblok-react/blob/main/lib/types.ts#L45
I'm wondering if the issue is coming directly from the storyblok-js-client lib. I'll talk to the team and come back later.

A workaround is to declare the type in a declaration file
// storyblok-rsc.d.ts
declare module '@storyblok/react/rsc' {
  export * from '@storyblok/react';
}

and then include it in tsconfig.json:
"include": ["storyblok-rsc.d.ts", "next-env.d.ts", ......]

@addemod I'm not sure I'm following what would that fix 🤔 I just tried and nothing happened

@addemod I'm not sure I'm following what would that fix 🤔 I just tried and nothing happened

The workaround is from OP in this issue: https://github.com/storyblok/storyblok-react/issues/570#issue-1749276890.
Not really a fix, as you are typing everything in the rsc module to use the typing from @storyblok/react.

Hello @carlonoelle @addemod!
We just released a new version of the React SDK that introduces some fixes we applied to storyblok-js-client.
https://github.com/storyblok/storyblok-react/releases/tag/v2.1.9
Can you check that types are working ok now for you?

Hey @fgiuliani and all,
Sorry i didn't respond, wenz on holiday right after submitting the issue.
Will check out the updated js client, and share if it fixed my bugs.

@fgiuliani Sadly I still have the same problem. /rsc and /bridge-loader are showing the same error as before:

I have @storyblok/react 2.1.9 installed. Also tried to forcefully install all packages again, npm install --force, still the same error.

@addemod did you apply the change mentioned here?
https://github.com/storyblok/storyblok-react/issues/570#issuecomment-1591207805
The fix that was added was related to the StoryblokClient methods (get, getStory, etc). Is that working now for you?

I could test it now, everything works for me. Only thing you have to do is setting moduleResolution to nodenext in your tsconfig, otherwise the import from @storyblok/react/rsc will show as unknown. I think this is your problem @addemod

@carlonoelle @fgiuliani My bad, I was confident that I already used nodenext, so I forgot to check it, but apparently I was still using node. It works as expected now :)

Good to hear, will close this now


Feature improve markdown

fix line-break and white-space handling
** ignore line-breaks being children of a tei:div
** ignore line-breaks due to indentation of the code; add a
** add additional line-breaks before lists
add item number to ordered lists
move to xsl computed-content generation instead of literal-results for the MD headers
add output methods for different output types
prettify XSLT indentation

sorry some of the commits are doubled due to some rebasing/merging I had to do and probably failed ;-) nevertheless the PR properly updates the XSLT that now will prettify some 1300 to 1400 files.

those should probably go to _guidelines-dev, shouldn't they?
or were that the files you moved by hand?

@lpugin please see comment above

Yes, these are the ones to move by hand. I just regenerated them and the MD is much better! Thanks


chore(deps): update terraform terraform-ibm-modules/key-protect/ibm to v2.7.0

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/key-protect/ibm (source)
module
minor
2.6.3 -> 2.7.0

Release Notes

terraform-ibm-modules/terraform-ibm-key-protect (terraform-ibm-modules/key-protect/ibm)
v2.7.0
Compare Source
Features

added new output key_protect_crn (#555) (6906496)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

:tada: This PR is included in version 2.4.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


labels file opening

Description
This path begins to add file opening functionality to our labels layer by exposing the path keyword arg on view_labels and add_labels. This will then get exposed in #616 or follow up PRs through our open dialog box.
Right now it just reuses the same file loading as with the image layer - which I think is fine. A question though is do we additionally want to do some type checking / casting of data into integers >= 0 (which is what our labels layer requires). If so, probably best to put it inside the init of Labels as no reason not to do that checking if someone just does add_labels(data), but I also don't want us to do unnecessary things that force computation when someone is say passing a massive dask array or zarr pyramid etc.
I know @jni and @perlman are currently working on #713 so curious how they think about this? If someone passes floats or negative numbers is that ok / their problem? If so then all this PR needs are tests (which could now be identical to the image IO tests / very simple).
@jni I also couldn't find an example of a labels like png or tif in skimage.data, do you know of a good one?
Type of change

[x] New feature (non-breaking change which adds functionality)

@sofroniewn no, I don't know that we have saved a labels image in skimage! One option would be to apply morphology.watershed(filters.sobel(data.coins()), markers=200) and then try a round trip.
I don't have a very strong opinion about forcing labels to be a specific type. Casting should only happen when viewing (ie in view_slice rather than in data itself) anyway, and should be fast. Negative labels is probably something someone will want at some point, and I'm ok with that. I'll less ok with float labels.

Since you asked for my 2¢, I suggest supporting any image type as a label channel. Labels are about uniqueness, and you are likely encounter {1,8,16,32,64)-bit (u)ints. If we can come up with a clean strategy for hashing a given data type, then it should be usable as a label.
Also, the current assumption that "0 is None" should be changed such that "arbitrary set of values are None", e.g., -1, MAX_INT, etc.

Thanks for weighing in @jni @perlman. I have left Labels such that it can still take anything, and agree we should behave gracefully with non-canonical input types, making sure that if we need to do casting we do it in the view or the vispy code can handle it.
@perlman I also agree moving towards "arbitrary set of values are None" is a good idea too, is that being enabled / make easier with #713 (feel free to respond over there / make a separate issue for this).
@jni I've added two simple round trip tests like what you describe but using blobs. I think this is ready for review now

I have now parameterized the test with suffix and will merge after the tests pass. I will follow-up on #470 in its own PR I can do that asap

The windows tests are failing because of how I use NamedTemporaryFile and permissions - see this warning here https://docs.python.org/3.6/library/tempfile.html#tempfile.NamedTemporaryFile
@jni @kne42 do either or you have any suggestions for how to fix this? stack overflow wasn't so helpful

@sofroniewn I think this issue is the reason why we created this in skimage:
https://github.com/scikit-image/scikit-image/blob/master/skimage/_shared/_tempfile.py
perhaps just use that? Even though it has an underscore so is theoretically private... It's been around for a long time and anyway, it's easy enough to roll our own if skimage's goes away.

I've added temporary_file.py - felt like something that better belonged in the repo, if you prefer it not there I can remove and just import the skimage one. We'll see if the tests now pass.
If you approve and the tests all pass can you then merge

I have dropped the public import and fixed the typo. If this looks good @jni you can merge after tests pass


Default metrics filters

In perf regression tests, there's plenty of metrics but only few are important for most use cases. Allow client code to decide important metrics, so filters are enabled for them - hiding from start non-important ones and speeding up drawing graphs.

@soyacz
with the new graph views it could be easy to select on as default, isn't it ?


JS-T: Search accounts

Search accounts
AC

[x] Search field to find accounts
[x] UI to display account names

PR https://github.com/steemit/devportal-tutorials-js/pull/83


[Open source] Issue that Exchangis1.0 query log is displayed incorrectly.

After clicking the query, this log is wrong, and it becomes the log of job. It should be the log of the task hello.
点击查询后这个日志不对，变成了job的日志，应该是hello这个task的日志

https://github.com/WeBankFinTech/Exchangis/pull/282，这个pr解决了


Error : Unable to log in Pluralsight.

[x ] I've verified and I assure that I'm running youtube-dl 2017.03.07

Before submitting an issue make sure you have:

[x ] At least skimmed through README and most notably FAQ and BUGS sections
[x ] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x ] Bug report (encountered problems with youtube-dl)

Hi, I have a problem, it say "ERROR: Unable to log in;". I try to download on Pluralsight. This my output ::
$ youtube-dl -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.03.07
[debug] Python version 2.7.12+ - Linux-4.8.0-39-generic-x86_64-with-Ubuntu-16.10-yakkety
[debug] exe versions: ffmpeg 3.0.7-0ubuntu<IP_ADDRESS>, ffprobe 3.0.7-0ubuntu<IP_ADDRESS>, rtmpdump 2.4
[debug] Proxy map: {}
Usage: youtube-dl [OPTIONS] URL [URL...]
And this it issue ::
$ youtube-dl --username<EMAIL_ADDRESS>--password ************ --write
-sub https://app.pluralsight.com/library/courses/aspdotnetcore-efcore-bootstrap-angular-web-app --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--username', u'PRIVATE', u'--password', u'PRIVATE', u'--write-sub', u'https://app.pluralsight.com/library/courses/aspdotnetcore-efcore-bootstrap-angular-web-app', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.03.07
[debug] Python version 2.7.12+ - Linux-4.8.0-39-generic-x86_64-with-Ubuntu-16.10-yakkety
[debug] exe versions: ffmpeg 3.0.7-0ubuntu<IP_ADDRESS>, ffprobe 3.0.7-0ubuntu<IP_ADDRESS>, rtmpdump 2.4
[debug] Proxy map: {}
[pluralsight:course] aspdotnetcore-efcore-bootstrap-angular-web-app: Downloading course JSON
[download] Downloading playlist: Building a Web App with ASP.NET Core, MVC 6, EF Core, and Angular
[pluralsight:course] playlist Building a Web App with ASP.NET Core, MVC 6, EF Core, and Angular: Collected 112 video ids (downloading 112 of them)
[download] Downloading video 1 of 112
[pluralsight] Downloading login page
[pluralsight] Logging in as<EMAIL_ADDRESS>ERROR: Unable to log in; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 761, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 426, in extract
self.initialize()
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 384, in initialize
self._real_initialize()
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/pluralsight.py", line 60, in _real_initialize
self._login()
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/pluralsight.py", line 100, in _login
raise ExtractorError('Unable to log in')
ExtractorError: Unable to log in; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Same problem here:
C:\Users\xxxx.xxxx\Downloads\Pluralsight - Angular 2 First Look>youtube-dl --username<EMAIL_ADDRESS>--password xxxxxxx --rate-limit 512K --verbose --sleep-interval 180 -o "%(autonumber)s - %(title)s.%(ext)s" --all-subs --playlist-start 1 https://app.pluralsight.com/library/courses/xxxxx-xx-xxxxx-xxxx
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', '--rate-limit', '512K', '--verbose', '--sleep-interval', '180', '-o', '%(autonumber)s - %(title)s.%(ext)s', '--all-subs', '--playlist-start', '1', 'https://app.pluralsight.com/library/courses/angular-2-first-look']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2017.03.07
[debug] Python version 3.4.4 - Windows-10-10.0.10586
[debug] exe versions: none
[debug] Proxy map: {}
[pluralsight:course] angular-2-first-look: Downloading course JSON
[download] Downloading playlist: Angular 2: First Look
[pluralsight:course] playlist Angular 2: First Look: Collected 95 video ids (downloading 95 of them)
[download] Downloading video 1 of 95
[pluralsight] Downloading login page
[pluralsight] Logging in as<EMAIL_ADDRESS>ERROR: Unable to log in; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvin3ute2\build\youtube_dl\YoutubeDL.py", line 761, in extract_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvin3ute2\build\youtube_dl\extractor\common.py", line 426, in extract
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvin3ute2\build\youtube_dl\extractor\common.py", line 384, in initialize
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvin3ute2\build\youtube_dl\extractor\pluralsight.py", line 60, in _real_initialize
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpvin3ute2\build\youtube_dl\extractor\pluralsight.py", line 100, in _login
youtube_dl.utils.ExtractorError: Unable to log in; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Same issue here for me too

Got the same error, w/ 2017.03.07 and 2017.02.x version.

Please, reply on this thread. Thanks.

https://github.com/rg3/youtube-dl/issues/12345#issuecomment-284054428


Support new grants.attack.bonus flags

https://gitlab.com/tposney/midi-qol/-/issues/662
Midi QOL 0.9.01 added support for some new flags detailed above. Consider adding support for them here.
Doesn't seem obvious how considering rollAttack's options doesn't take a random bonus like it does for advantage/disadvantage. Would probably have to generate parts and add the bonus to that.

I'm not looking to re-implement all the flags that Midi defines. It's mostly about the advantage/disadvantage reminders so this seems out of scope


Fix mergeabot config

Hm, in https://github.com/maxjacobson/seasoning/pull/1109 I copied the example from their README, but maybe it had an error, because I see this:
https://github.com/maxjacobson/seasoning/actions/runs/5093441480/workflow

Invalid workflow file
schedule accepts a list of one or more maps with the cron key set

Looking at the docs for schedule, it looks like this is supposed to be a list, which is consistent with what the error message says. https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule

FYI @pbrisbin ^^

Oops! Fixed in https://github.com/freckle/mergeabot-action/commit/92bd0caf95b90bfea64c7159dc37b1dca750c68f, thanks.

💪🏻 niceee. Excited to try this! Thanks for making it


image operating system "windows" cannot be used on this platform

Could not build and Windowsservercore based image using acr build.
Error message / details:
image operating system "windows" cannot be used on this platform
Error: failed to run step id: acb_step_0: exit status 1
Build ID: ad1 failed after 40.959416168s
Could you please advise whether windows containers are supported? If not, by when would it be?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c97db060-5c54-3b95-8382-59822b1d05b7
Version Independent ID: 2e1c320c-22c1-8edd-d19e-a483e04ea8ab
Content: Tutorial - Build container images in the cloud with Azure Container Registry Build
Content Source: articles/container-registry/container-registry-tutorial-quick-build.md
Service: container-registry
GitHub Login: @mmacy
Microsoft Alias: marsma

Thanks for the feedback! We are currently investigating and will update you shortly.

Hi @EriksonBahr, are you passing the --os parameter in your build command?
--os   : The operating system type required for the build.  Allowed values:
         Linux, Windows.  Default: Linux.

Hi
No, I didn't . I'll give it a try and let you know.

Worked! Thank you very much.

Excellent! @MicahMcKittrick-MSFT we can close this one--thanks.

--os deprecated, so --platform


LibreSSL question

Hi,
Thanks for enabling providing openssl-compatible library for Amiga!
I have a question regarding portability. Why openssl was chosen as a base when this implementation exists: https://github.com/libressl-portable/portable ? Would it be possible to incorporate all improvements over standard openssl tree into some patch and try to rebase it on top of libressl?

Probably the main reason is because AmiSSL was created almost 20 years old and LibreSSL did not exist then?
AmiSSL could in theory be extended to support different SSL libraries, but LibreSSL is forked of an old buggy OpenSSL base, whilst most of the improvements are already present in the latest OpenSSL. OpenSSL is more up to date with the latest standards and is more portable.


Log level change trace switch name #127

Changed TraceSwitchName to be different from the one in Web.config.
Removed duplicate timestamp in ping log.

excellent job @guo7711 . It worked as expected at my end. @bsvensson i will go ahead and merge.

Thank you @guo7711 @afili :)


Change Law payment status for orderlines

If folio order type Ongoing, payment status = "Ongoing"
For folio order type One-Time, then if date loaded = date: "Fully paid", if date loaded = NEVER: "Awaiting payment"
If folio acq method = gift (or Symphony order type GIFTMONO or GIFTSER): "Payment not required"

Not ready yet. Jake to confirm order that these things should be checked.

Confirmed the new mapping with Jake.


⚠ DesiTorrents has degraded performance

In a447fcd, DesiTorrents ($DST) experienced degraded performance:

HTTP code: 200
Response time: 3554 ms

Resolved: DesiTorrents performance has improved in be34927 after 8 minutes.


HDDS-4428. IT TestOzoneManagerHAMetadataOnly.testOMRetryCache is flaky

What changes were proposed in this pull request?
Fix the bug about org.apache.hadoop.ozone.om.TestOzoneManagerHAMetadataOnly.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4428
How was this patch tested?
In my original test, an error occurred on average 30 times,but now there are no more errors in several tests.

``

Thanx @ayushtkn @bharatviswa504 review. I agree with you that the issue has been resolved.So I will close this pr.


devbox: 0.13.0 -> 0.13.3

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 346260 run on x86_64-linux 1

1 package built:

devbox

Result of nixpkgs-review pr 346260 run on x86_64-linux 1

1 package built:

devbox

Could you please fix cross compilation:
diff --git i/pkgs/development/tools/devbox/default.nix w/pkgs/development/tools/devbox/default.nix
index b0432c609a8c..cc2e9a61e04a 100644
--- i/pkgs/development/tools/devbox/default.nix
+++ w/pkgs/development/tools/devbox/default.nix
@@ -2,6 +2,7 @@
 , fetchFromGitHub
 , installShellFiles
 , lib
+, stdenv
 }:
 buildGoModule rec {
   pname = "devbox";
@@ -27,7 +28,7 @@ buildGoModule rec {

   nativeBuildInputs = [ installShellFiles ];

-  postInstall = ''
+  postInstall = lib.optionalString (stdenv.buildPlatform.canExecute stdenv.hostPlatform) ''
     installShellCompletion --cmd devbox \
       --bash <($out/bin/devbox completion bash) \
       --fish <($out/bin/devbox completion fish) \

[nix-shell:~/.cache/nixpkgs-review/pr-346260]$ results/devbox/bin/updater
panic: lstat go.jetpack.io/devbox/examples: no such file or directory

goroutine 1 [running]:
main.main()
	go.jetpack.io/devbox/testscripts/testrunner/updater/main.go:19 +0x2b

Looks like subPackages = [ "cmd/devbox" ]; should be added.

@sikmir Thanks for the suggestions! I've amended the PR to include your feedback


Update hyper-rustls and fix lints

hyper-rustls now has a builder type and the old with_native_certs is gone.
HTTP 2 (h2) is now disabled by default.

Does that mean we are not using HTTP2 anymore when talking to splits.io? I think I'd prefer HTTP 2 over HTTP 1.

I'll add it back in then

Looks good now, thanks for working on this. 👍


Do not verify CSRF for every rodauth request

I'm having a problem integrating roda-oauth in a particular case where I need to allow a POST request to an /oauth-token URI; such requests are usually performed in the backend, and do not go through an html form, therefore the CSRF protection doesn't make sense.
This route is defined from inside the rodauth feature, however, rodauth-rails runs the verify_authenticity_token by default for every rodauth request, so it breaks my flow.
My suggestion would be to remove this line, or skip it given some circumstance around the request. The roda integration with csrf protection works, so there's something being done there that allows csrf protection to be skipped which hasn't been replicated in rodauth-rails.

We should probably add a #rails_check_csrf? method that defaults to true, which can then be overridden manually for specific routes.
Though I'm wondering what Roda's CSRF protection does differently 🤔 You either require a valid CSRF token for a route or not, it should also require that you explicitly disable CSRF protection for a certain route IMO. Maybe there is more to CSRF protection than I think.

roda's csrf protection is just rack_csrf. I'm gonna investigate this a bit more today, but the bottom line is, when working on rails, testing controllers and performing post calls, the token verification is somehow skipped. I'll have to find the how.

That's good to know, I thought you were using Roda's route_csrf plugin which doesn't use the rack_csrf gem. Thanks for investigating.

I think I found the issue (sort of).
rodauth-rails calls verify_authenticity_token inside the rodauth feature, as an extension to the csrf plugin; however, there is a rodauth controller, where this callback should/could be defined:
class RodauthController < ApplicationController
before_action :verify_authenticity_token
end

In order to make this more compliant with the rails way, rodauth should skip/disable the roda csrf plugin, and let the rodauth controller validate the authenticity token where rails expects it. Rails developers already know that before_action, and can choose to skip it if they desire so (or conditionally, as is my case).
Also, I found out that csrf protection is disabled in test mode by default, although for some reason that check doesn't pick up that variable.

How rodauth-rails works is that, before every action, the following happens:

#verify_authenticity_token is called on RodauthController
Rodauth action is performed (entirely in Roda)
when a template needs to be rendered, RodauthController is called for that

So, rodauth-rails doesn't execute "actions" on the controller, which means before_action callbacks are not applicable.
Note that rodauth-rails disables Roda's CSRF plugin: https://github.com/janko/rodauth-rails/blob/45ba96c3f1401ba92f0d13192c43b5439f1e19aa/lib/rodauth/rails/app.rb#L16
And in test mode the CRSF protection should be properly skipped, because #verify_authenticity_token calls #verified_request?, which checks #protect_against_forgery?, which read the allow_forgery_protection setting that's set to false in test environment. The rodauth-demo-rails app has some tests that work without CSRF protection.

I see now, thx for the details. The way I see it, it should then work like this:

Instead of "#verify_authenticity_token is called on RodauthController", all before_action callbacks from the rodauth controller should be run in before_rodauth (which means that all after_actions should run in after_rodauth. From the perspective of someone coming from rails development, they won't expect to define such callbacks in rodauth , but rather in rails controller.

or
rodauth-rails becomes a rails engine instead of a railtie. If you see how devise integrates with rails, all of the "rails low-level details" (devise controllers, views, etc) are hidden from the developer, until the developer explicitly requests them (i.e. there's a generator to copy devise views to the project main views directory). rodauth-rails could then provide similar functionality.
Of course, this is a totally separate "feature request", I guess :)
However, from the perspective of putting rails things in rails, I think that csrf protection should be altogether "removed" from rodauth and put into rails controls, as it's less of a burden for a rails developer, as the internet is full of "how do I disable csrf protection in rails" and this project would benefit of complying with the norm.
Let me know what you think.

The rodauth-demo-rails app has some tests that work without CSRF protection.

It's true, rails disables csrf protection in test mode by default. I managed to circumvent this by explicitly defining in application.rb:
config.action_controller.allow_forgery_protection = false

and it works now. However, my problem still stands (having rodauth routes that I want to skip csrf protection for).

Thank you for your input. I'll try to provide an answer for the two suggestions:
1⃣ I do agree the current behaviour is non-standard compared to other Rails gems, but I think that's a result of Rodauth's unique design, and I don't think it would be slippery slope to try to mask it into something it isn't. rodauth-rails doesn't run controller actions, so I don't think it's correct to try to run callbacks. Note that AFAIK there is no after_rodauth hook at the moment, so after_action couldn't really be implemented.
I also don't know how to call Action Controller callbacks separately from executing an action. I also bet that API might differ depending on the Rails version (rodauth-rails supports Rails 4.2+ at the moment). Note that the main reason rodauth-rails is using a separate controller is for rendering templates – I'd use Action View directly if it wasn't coupled to Action Controller. I tried rendering templates with ApplicationController, but view paths weren't being set correctly.
2⃣ I've explored making rodauth-rails a Rails engine, I was going back on forth when deciding on the design, but in the end I didn't see how it would benefit from being an engine, and I chose a simple railtie instead. I wanted everything to integrate automatically, but without hiding things that aren't necessary to hide. For example, I wanted to show copy Rodauth controller in the codebase for transparency.

If we provide instructions on disabling CSRF protection, I don't see a benefit in having it on the controller level. It would add additional maintenance burden, and IMO it would lead away from understanding how rodauth-rails works. Note that you can already disable CSRF protection:
rails_check_csrf! { super() unless request.path == oauth_token_path }

Thx, your response is pretty valid. I get most of the concerns, I'll leave a few more comments taking these into consideration:

rodauth-rails doesn't run controller actions, so I don't think it's correct to try to run Action Controller callbacks.

Agreed, better not go that route then.

but in the end I didn't see how it would benefit from being an engine, and I chose a simple railtie instead.

I'd argue that, if it were an engine, you'd manage to keep RodauthController for compatibility, without having to expose it to the user application, which is what happens now. I think what you want to achieve is allow a user to configure rodauth through rodauth_app, and the RodauthController is just there as "burden" to render views. Devise uses the same approach using engines, and I think rodauth-rails could benefit from it (same thing with the models/views: the moment you need them, you run the generator to load them under app, a la devise).

I wanted to show copy Rodauth controller in the codebase for transparency.

I just read this now. What do you mean by "for transparency"? I think that you'd gain from not exposing this detail to the user, for the possibility of him doing smth "rails-y", like before actions.

Note that you can already disable CSRF protection:

The thing is, I already do this here. rodauth-rails doesn't seem to use it, but rodauth defines it. Do you think it makes sense to use it in your case?

I understand what you mean. However, the transparency is useful when the user wants to change the directory of their templates, which requires renaming the Rodauth controller and changing rails_controller in the Rodauth config; this is straightforward when everything is already in your app.
It also allows the user to define additional helper methods via the current RodauthController, which they wouldn't be able to do if it was hidden inside an engine. I mean, they would have to define their own controller, but that might not be so obvious.
I have a comment in the generated Rodauth controller file that this controller is used only for rendering views, so I'm hoping that is enough.

The thing is, I already do this here. rodauth-rails doesn't seem to use it, but rodauth defines it. Do you think it makes sense to use it in your case?

Thank you for shedding light on this, rodauth-rails should definitely hook into #check_csrf?, I've opened a pull request that will make this possible – https://github.com/jeremyevans/rodauth/pull/96.

Thank you for shedding light on this, rodauth-rails should definitely hook into #check_csrf?, I've opened a pull request that should make this possible

Great, thx Janko, this will do perfect, thx for the patch!

I understand what you mean. However, the transparency is useful when the user wants to change the directory of their templates ...

Just to be clear, my suggestions are only based on what devise does and what rails users might expect or be confused with. Your concerns are perfectly valid to me. However, I can feel that you'd like rodauth-rails to become a worthy competitor to devise, which is still the rails auth standard, and in order to do so, it should achieve most of the goals that devise does, while doing everything else better. Those are, IMO:

quick-and-dirty bootstrap: you run rails generate devise:install + 1/2 declarations and you can see the forms there, with relatively few devise-specific code next to yours.
easy customization: I think rodauth-rails has a great shot at doing it way easier than devise.

Most of your concerns are amendable:

when the user wants to change the directory of their templates, which requires renaming the Rodauth controller and changing rails_controller in the Rodauth config...

True, but if rails_controller defaults to RodauthController, which lives in a rails engine, the user doesn't need to "understand" that command (and if he does, it is a customization step). Devise achieves this with DeviseController. It'd be easier to tell your user to rails_controller { YourController } and class YourController < RodauthController, thereby making RodauthController yours to tweak.

when the user wants to change the directory of their templates, which requires renaming the Rodauth controller and changing rails_controller in the Rodauth config

I'd argue that they could define a RodauthHelper for that.

I feel like making rodauth-rails an engine would be an overkill just to hide RodauthController...

Not only for that, it'd also "hide" your default views, thereby achieving that "quick-and-dirty bootstrap" and "customize as you go".
However, I like it already as is, so take my comments with a "speculative grain of salt", as I'm assuming that users will feel one way, but they might feel differently in the end, so this might already be the setup users want. Maybe it's worth to wait for a few more opinions and questions.
P.S.: A very useful thing for this project would be a "How to migrate from Devise to Rodauth-Rails" page :)

Yes, I would like this gem to be equally easy to work with as Devise, so it's great to hear any feedback.
I agree that RodauthController can be considered an implementation detail, however, if I were to hide it, I might need to add additional settings, such as parent_controller which Devise has (I guess not every Rails app has an ApplicationController? 🤷)
Yes, you're 100% percent correct regarding adding helper methods via RodauthHelper. I just thought that someone might be using some tricks to define helper methods only for views generated by a certain controller (because helper methods are global, the naming is just a convention). I don't know how to do it at the moment, but I thought if it's possible then it's probably configured on the controller level. It's a stretch for sure, though.

Not only for that, it'd also "hide" your default views, thereby achieving that "quick-and-dirty bootstrap" and "customize as you go".

Note that with rodauth-rails views already hidden, by default Rodauth's internal templates are rendered, until the user decides to run rails generate rodauth:views to copy the views into their app. With this I was hoping to achieve a similar customize-as-you-go approach that Devise has.

A very useful thing for this project would be a "How to migrate from Devise to Rodauth-Rails" page :)

Oh yeah, I haven't considered this, it sounds very useful indeed 👍 Personally I'm not experienced with Devise (I've created rodauth-rails because I didn't feel like learning Devise and then figuring out it's too complex for me 😛), but I will try to write a guide at some point.

Note that AFAIK there is no after_rodauth hook at the moment, so after_action couldn't really be implemented.

To actually support the full range of callbacks, there would need to be an around_rodauth. With such a hook point, the implementation to run all callbacks would actually be very straight forward:
around_rodauth do
  rails_controller_instance.run_callbacks(:process_action) do
    yield # or super(), depending on how `around_rodauth` would be defined
  end
end

That would be sufficient plus perhaps disabling the Roda-owned CSRF check or adding skip_before_filter in the Rodauth controller, so that this check isn't doubled up.
I expended some energy trying to do this myself with a little bit too much cleverness, but ultimately was foiled by the Rodauth lifecycle and when the route handlers are actually designed:
methods.grep(/^_handle_/).each do |route_handler|
  define_method(route_handler) do
    rails_controller_instance.run_callbacks(:process_action) do
      super()
    end
  end
end

Interesting, thank you for sharing. The problem with implementing an around or after callback like this is that it won't work with redirects or early responses, because Rodauth uses throw for that, which would exit from any around block.
I think the correct way to do this would be to utilize Roda's hooks plugin. I'll see if it's possible to add around hooks there, and then possibly add after_rodauth and around_rodauth hooks that use that. Or maybe it's somehow possible to add them directly to Rodauth 🤔

@bjeanes Actually, do you think you could make a proposal for the around_rodauth hook to Jeremy on the Rodauth google group? I realized it would be complex to start from Roda like I suggested, and that throwing will still execute any ensure blocks (which I'm assuming #run_callbacks uses), so it doesn't sound like it would be difficult to do inside Rodauth after all.

Actually, do you think you could make a proposal for the around_rodauth hook to Jeremy on the Rodauth google group?

I had a reminder to come back to this but then work got a bit busy. I will do that today, but first I'm going to have a short play in rodauth code-base to see if I can just implement at least a proof-of-concept for it within my window I have for this today.

Here is a proof-of-concept for the around_* hooks as a general purpose utility for any Rodauth feature to use. In the PoC, I switched from before_rodauth to around_rodauth handling.
https://github.com/jeremyevans/rodauth/compare/master...bjeanes:around_rodauth?expand=1
I'll email Jeremy on the mailing list now to discuss whether he'd be interested in moving in this direction.

https://groups.google.com/g/rodauth/c/kZfPJdVdQI8

https://github.com/jeremyevans/rodauth/pull/129

@HoneyryderChuck I'm preparing to release https://github.com/janko/rodauth-rails/commit/2ca331ea94e5daeea910cc2a41c7dc77c7b2c02c, which would run Action Controller action callbacks and rescue handlers around Rodauth actions. Because now CSRF token would be verified automatically as part of running controller action callbacks, I've removed explicit CSRF protection and overrides of #check_csrf and #check_csrf? that I had.
However, I think this can be problematic when integrating rodauth-oauth, because it overrides #check_csrf? to skip CSRF protection in some cases. With this new change that wouldn't work, because now the Rails controller decides when the CSRF token is verified, not the #check_csrf? method, so by default CSRF token would always be verified.
I'm not sure exactly how to solve this problem, I thought you might have ideas. My only idea was adding something like:
skip_before_action :verify_authenticity_token, unless: -> { rodauth.check_csrf? }

to the Rodauth controller, but the problem is this won't be possible for existing users of rodauth-rails, as their Rodauth controller has already been generated. That problem probably wouldn't exist if rodauth-rails was an engine like you initially suggested, though I still think that would bring its own challenges. Maybe there is another solution I'm not seeing.

@janko What about calling rails_controller_instance.skip_forgery_protection if check_csrf? is true. That won't require a modification from the controller. The downside is its hidden away and for a first-timer looking at it may seem backwards: "skip csrf if check_csrf?" will seem inverted. That could be mitigated by good naming I suppose...

I'll also note that performing CSRF protection twice may not be horrendous in most cases (personally, I emit a bunch of metrics that woudl be doubled up so it would be PITA, but I think for most it'll just be the wasted clock time)

@janko thanks for the heads-up! This would have gone under my radar otherwise.

That problem probably wouldn't exist if rodauth-rails was an engine like you initially suggested,...

I guess that one came back to haunt you :) eheh . For the record, although I still think an engine would be more appropriate, this was just one of the reasons.
I don't think one can do much about already existing controllers, besides documentation and deprecation warnings. Would a rodauth controlller callback be able to check for present of both rodauth-oauth and lack of a verify_authenticity_token declaration for RodauthController suffice? Because I'm assuming that existing applications have this optionn turned off, so you want them to turn it on.
However, when turning it on, maybe it's best to had a filter for skipping it for JSON requests? Something in the lines of:
skip_before_action :verify_authenticity_token, unless: -> { request.accepts_json? }

I guess you'd want this in the generator altogether, to make using rodauth-rails in JSON mode work out of the box?

@bjeanes I like this idea. I wasn't sure whether CSRF token verification can be skipped at the controller instance level, but the allow_forgery_protection boolean attribute works on both class and instance level. So, if we add back #check_csrf and #check_csrf? overrides, then we could have Rodauth call into #verify_authenticity_token manually like before, and then when running the callbacks we could disable CSRF protection:
def rails_controller_callbacks
  rails_controller_instance.allow_forgery_protection = false
  rails_controller_instance.run_callbacks(:process_action) do
    yield
  end
end

I think this should work nicely, because Rodauth calls #check_csrf before the new #_around_rodauth. The #check_csrf? should already default to false for JSON requests, as that's the case for Rodauth's jwt feature.

Makes sense to me!


Create test for run operator execute

Test execution of run operator with correct parameters.
Add a new global variable TEST_DATA_PIPELINE_NAME to use in the run operator.
Removed line 31 importing DataPipelineHook. Sine we are using mocks for the hook, we don't need to import the real hook.

will redo pr with finished tests


[1164] Sync.poll doesn't retry if the request fails for some reason :…

… bug fixed
Fixes #1164
Description
Sync.poll doesn't retry if the request fails for some reason.
Type
Choose one: Bug fix
Checklist

[x] I have read and acknowledged the Code of conduct
[x] I have read the Contributing page
[x] I have signed the Google Individual CLA, or I am covered by my company's Corporate CLA
[x] I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach
[x] I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the style guide of this project
[x] I have run ./gradlew check and ./gradlew connectedCheck to test my changes locally
[x] I have built and run the reference app(s) to verify my change fixes the issue and/or does not break the reference app(s)

@PallaviGanorkar Please use Description section above to describe your solution.

@PallaviGanorkar Please use Description section above to describe your solution.

@PallaviGanorkar Please use Description section above to describe your solution.

Updated Description

@PallaviGanorkar , are the changes ready so that @aditya-07  can review?

@aditya-07  Please review the code changes, As discussed, reusing existing method for periodic sync with retry mechanism

@PallaviGanorkar Please use Description section above to describe your solution.

done

Please add appropriate tests to check that the retry mechanism is working.

@aditya-07  As discussed, Code changes are tested Manually, those are working fine.
Test case will be pushed as separate PR under issue https://github.com/google/android-fhir/issues/1252

As discussed, Code changes are tested Manually, those are working fine.
Test case will be pushed as separate PR under issue #1252

As discussed, Code changes are tested Manually, those are working fine.
Test case will be added as separate PR under issue https://github.com/google/android-fhir/issues/1252

Just one small cleanup, otherwise LGTM!

Done


[Dev][Footer]Fix footer links to unrouted pages

Beginner, Intermediate, and Advanced Tips are products of a filter applied to the content; there are no pages or routes that could take the user directly to those pages.
But, the current footer design contemplates those links.
Can you create links that take the users directly to each of those selections? Beginner, Intermediate, and Advanced Tips to fix the current footer links?
This issue aims to solve this issue either finding a solution to enable routering the links to a tips page displaying cards with only the selected tips level or proposing a new design compatible with the dev possibilities.

This was implemented.


Port not configurable?

Port not configurable? Only 3306? Can't it be another port?

It is, see the readme: https://github.com/bradzacher/mysqldump#options


Error when flashing ESP32-S3 DevKit (IDFGH-8767)

Error occurs when flashing ESP32-S3-DevKitC-1 when using master branch:
...
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
A fatal error occurred: bootloader/bootloader.bin requires chip revision in range [v0.0 - v0.99] (this chip is revision v2.8). Use --force to flash anyway.

This issue will be solved by the new esptool release (very soon).

esptool v4.4 has just been released (https://github.com/espressif/esptool/releases/tag/v4.4). It should contain the fix for this. Please re-run the install script and it should be updated to this version.


ServiceID needed as ENV variable

Hi there,
We have a small feature request.
Registrator is registering in consul with specific ServiceID:
ID    string               // <hostname>:<container-name>:<internal-port>[:udp if udp]
It would be very helpful, if this ServiceID was available as ENV variable (SERVICE_ID or whatever)
I know, that all those ServiceID pieces (container-name / internal port) are possible to get manually,
with docker command (or docker API) but then we will have to allow each container to access docker socket (or docker API) which is IMHO too much (security issues), or uncomfortable or too heavy when you have a minimalistic containers.
Why we need ServiceID ?
When we register into consul, it is automatically getting into Load Balancer (HAproxy etc.).
But when we stop a container it should be de-registered first form consul (automatically also from LB) and then destroyed. For manual deregistering we need ServiceID.
Hopefully docker stop  is sending SIGTERM, and SIGTERM we can easily trap inside container and de-register a service from consul before dying.
like in example here:
https://github.com/sielaq/paas-in-a-box/blob/master/examples/SimpleWebappPython/simple_webapp_python/start.sh#L14
Is it technically possible to add ServiceID as ENV variable, so we do not have to do container-name/port discovery?
Any other ideas are very welcome :)

I don't understand. You want to manually deregister services from the
container to deregister... why not let Registrator do this for you?
On Mon, Feb 23, 2015 at 3:37 AM, Wojciech Sielski<EMAIL_ADDRESS>wrote:

Hi there,
We have a small feature request.
Registrator is registering in consul with specific ServiceID:
ID string // ::[:udp if udp]
It would be very helpful, if this ServiceID was available as ENV variable
(SERVICE_ID or whatever)
I know, that all those ServiceID pieces (container-name / internal port)
are possible to get manually,
with docker command (or docker API) but then we will have to allow each
container to access docker socket (or docker API) which is IMHO too much
(security issues), or uncomfortable or too heavy when you have a
minimalistic containers.
Why we need ServiceID ?
When we register into consul, it is automatically getting into Load
Balancer (HAproxy etc.).
But when we stop a container it should be de-registered first form
consul (automatically also from LB) and then destroyed. Hopefully docker
stop is sending SIGTERM, and SIGTERM we can easily trap inside container
and de-register a service from consul before dying.
like in example here:
https://github.com/sielaq/paas-in-a-box/blob/master/examples/SimpleWebappPython/simple_webapp_python/start.sh#L14
Is it technically possible to add ServiceID as ENV variable, so we do not
have to do container-name/port discovery?
Any other ideas are very welcome :)
—
Reply to this email directly or view it on GitHub
https://github.com/gliderlabs/registrator/issues/122.

--
Jeff Lindsay
http://progrium.com

Because Registrator is doing it after container is gone, so for a small amount of time it is still in consul and HAproxy loadbalancer (I got 503 error for some time). So I need to deregister it before container is gone, so no one will notice that.

How long were you getting 503s? What are you using to update HAproxy?
On Mon, Feb 23, 2015 at 7:49 AM, Wojciech Sielski<EMAIL_ADDRESS>wrote:

Because Registrator is doing it after container is gone, so for a small
amount of time it is still in consul and HAproxy loadbalancer (I got 503
error for some time). So I need to deregister it before container is
gone, so no one will notice that.
—
Reply to this email directly or view it on GitHub
https://github.com/gliderlabs/registrator/issues/122#issuecomment-75543147
.

--
Jeff Lindsay
http://progrium.com

I got 503 for around half second.
I'm using consul-template for HAproxy updates.
when I do with manual deregistering (like mentioned) I got 0 downtime, so consul-template works fine.

moreover this is half second per backend (instance), so if I a do a rolling update of a service with 20 backends (instances) - I got 0.5sec x 20 of different downtimes.

You're going to have this problem regardless. Especially in failure
scenarios. Seems like you should be optimizing for those instead of ideal
scenarios like this. Try using tags or a switch over strategy that will let
you keep running old instances so you can stop them afterwards.
Alternatively you can probably reconfigure or redesign your balancer to
better tolerate these situations.
On Mon, Feb 23, 2015 at 8:02 AM, Wojciech Sielski<EMAIL_ADDRESS>wrote:

moreover this is half second per backend (instance), so if I a do a
rolling update of a service with 20 backends (instances) - I got 0.5sec x
20 of different downtimes.
—
Reply to this email directly or view it on GitHub
https://github.com/gliderlabs/registrator/issues/122#issuecomment-75545119
.

--
Jeff Lindsay
http://progrium.com

True, everybody need to deal with failure scenarios - there is no doubt about that.
I'm talking about daily normal usage.
We can't use other method / deployment strategy - currently we use Mesos and Marathon for that, (which has limited method of rolling updates) until something more mature and robust gonna come out.
What I want to say is, having this ServiceID as environment variable would help us much.

I understand. However, I'm hesitant to add this because I don't think it's
the right way to customize ServiceID and I'd rather not address that as a
problem until I understand the use case. I want to avoid short term fixes
that add to the API and rather find other workarounds in how other
components are configured. Otherwise people may rely on it and it will be
harder to remove or change.
But thank you for explaining the situation. Maybe we'll take this route,
but not without some thought. I'd also like to see if you can solve this
another way.
On Mon, Feb 23, 2015 at 8:40 AM, Wojciech Sielski<EMAIL_ADDRESS>wrote:

True, everybody need to deal with failure scenarios - there is no doubt
about that.
I'm talking about daily normal usage.
We can't use other method / deployment strategy - currently we use Mesos
and Marathon for that, (which has limited method of rolling updates) until
something more mature and robust gonna come out.
What I want to say is, having this ServiceID as environment variable would
help us much.
—
Reply to this email directly or view it on GitHub
https://github.com/gliderlabs/registrator/issues/122#issuecomment-75551706
.

--
Jeff Lindsay
http://progrium.com

Don't get me wrong.
I don't want to customize SeviceID, I just need to have it available as ENV variable. So the same ID which is used by Registator (to register into consul)  I need to use for deregistering in case of SIGTERM.
I have already shown you an example how I can solve it:
https://github.com/sielaq/paas-in-a-box/blob/master/examples/SimpleWebappPython/simple_webapp_python/start.sh#L14
which is very heavy solution since I need docker inside container.

Oh. But I don't know how you'd do that without setting it. Registrator
can't set environment on containers. Registrator only reads it, which is
why I thought you wanted to change/set it via environment...
On Mon, Feb 23, 2015 at 8:59 AM, Wojciech Sielski<EMAIL_ADDRESS>wrote:

Don't get me wrong.
I don't want to customize SeviceID, I just need to have it available as
ENV variable. So the same ID which is used by Registator (to register into
consul) I need to use for deregistering in case of SIGTERM.
I have already shown you an example how I can solve it:
https://github.com/sielaq/paas-in-a-box/blob/master/examples/SimpleWebappPython/simple_webapp_python/start.sh#L14
which is very heavy solution since I need docker inside container.
—
Reply to this email directly or view it on GitHub
https://github.com/gliderlabs/registrator/issues/122#issuecomment-75556298
.

--
Jeff Lindsay
http://progrium.com

ok, technically it is not possible :) this is what I was suspecting :)
I just wanted this confirmation.
Then I have to live with my workapnurd :) but thx anyway!
topic closed

Just an easy and better workaround,
for someone who would be interested:
I have created a dummy wrapper for docker (use by Mesos Slave),
so Mesos container name is injected as ENV variable.
( Based on Container Name you can easily create Registrator Consul ID
like example here )
https://github.com/eBayClassifiedsGroup/paas-in-a-box/blob/master/infrastructure/mesos-slave/docker
and I'm not de-registering anymore - I leave it for Registrator,
I'm using maintenance mode - new feature of consul - so in the end,
it works in the same way - is being taken out from Load Balancer before stopping.
(Specially needed for Mesos+Marathon rolling updates)


Scaler not working

Hi,
Thanks for the project - and thanks for the many enhancements recently.  I'm having trouble getting the autoscaler working.  I just built the docker image and ran it with:
docker run -e MESOS_URI=http://x.x.x.x:5050 -e MARATHON_URI=http://x.x.x.x:8080 -e AGENT_PORT=5051 marathon-autoscaler
From the startup logs, it looks like everything is running ok:
2016-10-21 17:09:26,543 INFO supervisord started with pid 1
2016-10-21 17:09:27,547 INFO spawned: 'marathon_autoscaler' with pid 7
2016-10-21 17:09:28,142 | INFO | Namespace(agent_port=5051, cpu_fan_out=None, datadog_api_key=None, datadog_app_key=None, datadog_env=None, enforce_version_match=False, log_config='/app/logging_config.json', marathon_pass=None, marathon_uri='http://x.x.x.x:8080', marathon_user=None, mesos_uri='http://x.x.x.x:5050', rules_prefix='mas_rule', sleep_interval=5)  (<module>:92)
2016-10-21 17:09:28,144 | INFO | Mesos and Marathon Connections Established.  (start:157)
2016-10-21 17:09:29,147 INFO success: marathon_autoscaler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-10-21 17:09:34,543 | INFO | Stats differentials collected.  (poll:86)
2016-10-21 17:09:34,544 | INFO | Decision process beginning.  (decide:56)
2016-10-21 17:09:34,545 | INFO | Decisions are completed.  (start:165)
2016-10-21 17:09:45,836 | INFO | Stats differentials collected.  (poll:86)
2016-10-21 17:09:45,837 | INFO | Decision process beginning.  (decide:56)
2016-10-21 17:09:45,837 | INFO | Decisions are completed.  (start:165)
2016-10-21 17:09:57,089 | INFO | Stats differentials collected.  (poll:86)

However, the app never scales under load with the following labels (using small durations for testing):
  "labels": {
  "use_marathon_autoscaler": "true",
  "min_instances": "1",
  "max_instances": "10",
  "mas_rule_fastscaleup_1": "cpu | >10 | PT10S | 3 | PT1M30S",
  "mas_rule_fastscaleup_2": "memory | >50 | PT10S | 3 | PT1M30S"
  }

Do you see any reason why this wouldn't work?

I also wanted to mention that I am using Mesos 1.0.1 and Marathon 1.3.2.

Change your use_marathon_autoscaler:
"labels": {
  "use_marathon_autoscaler": "0.0.3",
  "min_instances": "1",
  "max_instances": "10",
  "mas_rule_fastscaleup_1": "cpu | >10 | PT10S | 3 | PT1M30S",
  "mas_rule_fastscaleup_2": "memory | >50 | PT10S | 3 | PT1M30S"
}

TL;DR;
We originally used use_marathon_autoscaler as just a boolean to check for an application's participation. Later, when we began improving the thresholds capabilities to be aware of the newer format you're using, we look for the version number where we introduced the newer thresholds format. We still accept a boolean value, but the autoscaler assumes you're using the older thresholds format if you use true. This was done so original applications could keep their older format and the autoscaler would be backward compatible.
Just for reference:
We have not yet moved our Mesos clusters to that new version you are running, but in the future you could place a modified logging_config.json on your Marathon agent hosts, put a volume in the app definition for the autoscaler that points to this json file and run the autoscaler with the LOG_CONFIG environment variable set to the container path to this modified logging configuration.
In the logging_config.json, set /root/level to DEBUG:
    "root": {
        "level": "DEBUG",
        "handlers": ["console"]
    }

This will give you more insight with the decision making process and I believe we also remark on which version of the thresholds it plans to look for based on the use_marathon_autoscaler value.

hi @kernelpanek-tendril - thanks for the response.
I enabled the DEBUG logging and changed the labels as you suggested, but I'm not getting much back.  Perhaps my version of Mesos?
2016-10-27 20:54:09,694 | INFO | Namespace(agent_port=5051, cpu_fan_out=None, datadog_api_key=None, datadog_app_key=None, datadog_env=None, enforce_version_match=False, log_config='/logging-config.json', marathon_pass=None, marathon_uri='http://<IP_ADDRESS>:8080', marathon_user=None, mesos_uri='http://<IP_ADDRESS>:5050', rules_prefix='mas_rule', sleep_interval=5)
2016-10-27 20:54:09,695 | INFO | Mesos and Marathon Connections Established.
2016-10-27 20:54:09,811 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:09,847 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:09,914 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:09,935 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:10,938 INFO success: marathon_autoscaler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-10-27 20:54:13,015 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:13,065 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:13,147 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:13,170 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:16,174 | INFO | Stats differentials collected.
2016-10-27 20:54:16,175 | INFO | Decision process beginning.
2016-10-27 20:54:16,175 | INFO | Decisions are completed.
2016-10-27 20:54:21,280 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:21,313 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:21,391 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:21,493 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:24,571 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:24,611 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:24,651 | DEBUG | (u'<IP_ADDRESS>', [])
2016-10-27 20:54:24,721 | DEBUG | (u'<IP_ADDRESS>', [])

it continues on like that, without ever changing...

We don't have that combination of Mesos/Marathon to test with, but based on what I see in your log (rather what I don't see) is the agent data... Each agent is coming back with nothing for its statistics. Each of those new DEBUG lines is showing an empty list for the call when it's querying the /monitor/statistics path of each agent.
If you can ssh to the agent where the autoscaler is running, try to curl the path on one of the other agents:
curl http://<IP_ADDRESS>:5051/monitor/statistics

Does it timeout, respond with something other than HTTP 200, or simply give back no data? My instincts tell me that we grossly underestimated the number of ways we can all configure Mesos and Marathon and our code does not accommodate for all of those possibilities, yet.

Looks like I get a 200, but nothing there:
root@03:~# curl -v http://<IP_ADDRESS>:5051/monitor/statistics
*   Trying <IP_ADDRESS>...
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 5051 (#0)
> GET /monitor/statistics HTTP/1.1
> Host: <IP_ADDRESS>:5051
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 28 Oct 2016 19:11:21 GMT
< Content-Length: 2
< Content-Type: application/json
<
* Connection #0 to host <IP_ADDRESS> left intact
[]

Do you run multiple instances of Mesos agents on the same hosts where the path would be?:
curl -v http://<IP_ADDRESS>:5051/slave(1)/monitor/statistics

No, just a single Mesos agent per host.

I'm not sure why your agents are not posting their executor stats, but I'm curious enough to find out how Mesos ends up in the configuration where they stop reporting executor statistics.
Just out of curiousity, does your Mesos master web UI report the CPU and memory usage of executors? When I wrote this app, I used the same approach the (web UI) Javascript takes to calculate CPU and memory usage.

hi @kernelpanek
I'm happy to try and help troubleshoot...  looks like the Mesos agents are reporting the correct number of CPUs and Memory:

Click on one of the agent ID links and that should bring up the page that I'm thinking does dynamic updates on CPU/memory utilization. The utilization shows on the left-hand side of the page.

oh you're right - weird!  the resource usage shows 0

Are you running on your own physical hardware or VMs? I'm grasping at straws here, but I'm not sure how the new resource isolators work in Mesos. The older versions of Mesos don't show the network statistics in the /monitor/statistics endpoint unless the isolator is enabled.  The difference is that the basic cpu and memory metrics still show.  I've seen screenshots of other Mesos setups where the number of individual metrics outnumbers what is shown in our Mesos clusters.  The last time I tried to hunt down how we could get our Mesos cluster to report those better details, I found myself slumped over in my couch from staying up all night reading through cgroup C++ code..  Various installations can have wildly inconsistent statistics pages.

I'm running on VMs in a a VPS.  It's strange that I can't get anything out of the /monitor/statistics endpoint, but I am able to get metrics out of /metrics/snapshot.  Would this endpoint provide enough detail to use the autoscaler?
I have checked with other Mesos users and have heard similar reports of empty responses using the /monitor/statistics endpoint.

Unfortunately, /metrics/snapshot only gives the high-level stats of the agent instance as a whole and in order to know a marathon application's usage or impact on that agent, we want the application performance details which is in the /monitor/statistics endpoint. The /state endpoint would be good if it actually gave utilization information and not just allocation information.
Even though your endpoint gives back an HTTP 200, are you using authentication on your agents? I'm wondering if this story was implemented in such a way that the endpoint still responds with a HTTP 200, but gives up nothing if no authentication header info is present.  If that's the case, I'll gladly accept your PR if you implement auth for agent endpoint querying.
Ultimately, I'm going to open an issue to incorporate testing with multiple mesos/marathon versions using Dell/Emc's vagrant repo and see if we can squeeze this into TravisCI somehow.

Hi @kernelpanek-tendril.  I was able to get to the bottom of my problem...
I forgot to mention that I am running Mesos from within a Docker container.  I started looking at the logs when I made the API request and saw errors like:
Failed to get resource statistics for executor 'sysdig-agent.1951556a-9ad9-11e6-a24a-0242aec327c9' of framework efaaca88-e937-4288-b929-2a0bd940e70a-0000: Failed to collect cgroup stats: Failed to determine cgroup for the 'cpu' subsystem: Failed to read /proc/2653/cgroup: Failed to open file: No such file or directory

...even though I had /proc mapped to /host/proc.  I was under the impression that Mesos could read from /host/proc, but that's not the case.  In order to get Mesos to find the process IDs, I had to run the Mesos container with --pid=host.
Thanks for the help!  I will start digging into the autoscaler now.  :)

@kernelpanek-tendril:
I got the /monitor/statistics endpoint working, however, the app never scales under load with the following labels (using small durations for testing):
  "labels": {
  "use_marathon_autoscaler": "0.0.3",
  "min_instances": "1",
  "max_instances": "10",
  "mas_rule_fastscaleup_1": "cpu | >10 | PT10S | 3 | PT1M30S",
  "mas_rule_fastscaleup_2": "memory | >50 | PT10S | 3 | PT1M30S"
  }

I can see in the logs that mem usage and cpu usage is above what I specified.
2016-11-01 23:23:46,205 | INFO | Stats differentials collected.
2016-11-01 23:23:46,207 | INFO | Decision process beginning.
2016-11-01 23:23:46,208 | DEBUG | Version matching is enforced. Version: 0.0.3
2016-11-01 23:23:46,208 | DEBUG | content-providers: participating
2016-11-01 23:23:46,208 | INFO | content-providers: metrics: {'mem': 28.426930581835386, 'cpu': 49.20513632498708}
2016-11-01 23:23:46,208 | INFO | content-providers: last_triggered_rule set to: None
2016-11-01 23:23:46,208 | INFO | content-providers: vote: 0 ; scale_factor requested: 0
2016-11-01 23:23:46,209 | INFO | content-providers: application ready: True
2016-11-01 23:23:46,209 | INFO | content-providers: instances: min:1, running:1, max:10
2016-11-01 23:23:46,211 | DEBUG | Version matching is enforced. Version: 0.0.3
2016-11-01 23:23:46,212 | DEBUG | content-providers: participating
2016-11-01 23:23:46,213 | INFO | Decisions are completed.

Any ideas?

Your metric:
2016-11-01 23:23:46,208 | INFO | content-providers: metrics: {'mem': 28.426930581835386, 'cpu': 49.20513632498708}

Your fastscaleup rule:
  "mas_rule_fastscaleup_1": "cpu | >10 | PT10S | 3 | PT1M30S",
  "mas_rule_fastscaleup_2": "memory | >50 | PT10S | 3 | PT1M30S"

What your rule is set up for is: "If CPU utilization is greater than 10% and memory utilization is greater than 50% for 1 minute, 30 seconds; then scale the application +3 instances." However, your memory usage has not reached that threshold.
If you are wanting the application to scale on either memory or CPU, then split the rule by providing different rule names, like so:
  "mas_rule_fastcpuscaleup_1": "cpu | >10 | PT10S | 3 | PT1M30S",
  "mas_rule_fastmemscaleup_1": "memory | >50 | PT10S | 3 | PT1M30S"

Thanks @kernelpanek-tendril
I adjusted the rules as you suggested, and was able to trigger the autoscaler to trigger.  However, I never see new instances, despite seeing scale_factor requested: 3 in the logs..
2016-11-02 04:24:46,137 | INFO | Decision process beginning.
2016-11-02 04:24:46,138 | DEBUG | Version matching is enforced. Version: 0.0.3
2016-11-02 04:24:46,138 | DEBUG | content-providers: participating
2016-11-02 04:24:46,138 | INFO | content-providers: metrics: {'mem': 16.858591532639547, 'cpu': 36.4259688760453}
2016-11-02 04:24:46,138 | INFO | content-providers: last_triggered_rule set to: [{'ruleInfo': {'rulePart': u'1', 'ruleName': u'fastcpuscaleup'}, 'ruleValue': {'scale_factor': u'3', 'weight': 1.0, 'threshold': {'val': u'10', 'op': u'>'}, 'metric': u'cpu', 'tolerance': u'PT10S', 'backoff': u'PT1M30S'}}]
2016-11-02 04:24:46,138 | INFO | content-providers: vote: 1 ; scale_factor requested: 3
2016-11-02 04:24:46,139 | INFO | content-providers: application ready: True
2016-11-02 04:24:46,139 | INFO | content-providers: instances: min:1, running:1, max:10
2016-11-02 04:24:46,139 | INFO | content-providers: tolerance window filled: True / 04:24:36.139353
2016-11-02 04:24:46,139 | DEBUG | content-providers: tolerance window filled: True / 04:24:36.139353
2016-11-02 04:24:46,139 | INFO | content-providers: within backoff window: False / 04:23:16.139694 - 04:24:46.139694
2016-11-02 04:24:46,139 | DEBUG | content-providers: []
2016-11-02 04:24:46,140 | DEBUG | Version matching is enforced. Version: 0.0.3
2016-11-02 04:24:46,140 | DEBUG | content-providers: participating
2016-11-02 04:24:46,140 | INFO | Decisions are completed.

What happens here?  DEBUG | content-providers: []

That line with the empty list is the past decisions in the backoff window (or timeframe). No items should result in returning False with regard to checking if the system is within the backoff period.  However, that isn't the problem.  I'm concerned with there not being any log entry for "tolerance reached: {True/False}" before the backoff window check, but after a positive "tolerance window filled: True" entry. I see the issue with the code on why this happens, but the tolerance has not yet been met because the performance tail has nothing in it. I'll submit a fix for this: #19.

hi @kernelpanek-tendril
Looks like tendrilinc/marathon-autoscaler/pull/21 was merged, but I'm still having trouble with my application scaling.
Here's my rule:
  "labels": {
      "use_marathon_autoscaler": "0.0.3",
      "min_instances": "1",
      "max_instances": "10",
      "mas_rule_fastcpuscaleup_1": "cpu | >10 | PT10S | 3 | PT30S"
  }

And here is the log from the autoscaler when I've got load on itL
2016-11-28 19:52:41,534 | INFO | Stats differentials collected.
2016-11-28 19:52:41,536 | INFO | Decision process beginning.
2016-11-28 19:52:41,537 | DEBUG | Version matching is enforced. Version: 0.0.3
2016-11-28 19:52:41,538 | DEBUG | content-providers: participating
2016-11-28 19:52:41,538 | INFO | content-providers: metrics: {'mem': 25.595185250946972, 'cpu':<PHONE_NUMBER>867287}
2016-11-28 19:52:41,539 | INFO | content-providers: last_triggered_rule set to: [{'ruleInfo': {'rulePart': u'1', 'ruleName': u'fastcpuscaleup'}, 'ruleValue': {'scale_factor': u'3', 'weight': 1.0, 'threshold': {'val': u'10', 'op': u'>'}, 'metric': u'cpu', 'tolerance': u'PT10S', 'backoff': u'PT30S'}}]
2016-11-28 19:52:41,540 | INFO | content-providers: vote: 1 ; scale_factor requested: 3
2016-11-28 19:52:41,540 | INFO | content-providers: application ready: True
2016-11-28 19:52:41,540 | INFO | content-providers: instances: min:1, running:1, max:10
2016-11-28 19:52:41,541 | INFO | content-providers: tolerance window filled: True / 19:52:31.541472
2016-11-28 19:52:41,542 | DEBUG | content-providers: tolerance window filled: True / 19:52:31.541472
2016-11-28 19:52:41,542 | INFO | content-providers: tolerance reached: False / 19:52:31.541472 - 19:52:41.541472
2016-11-28 19:52:41,542 | DEBUG | content-providers: vote_list: []
2016-11-28 19:52:41,543 | INFO | content-providers: within backoff window: False / 19:52:11.543266 - 19:52:41.543266
2016-11-28 19:52:41,543 | DEBUG | content-providers: []
2016-11-28 19:52:41,544 | DEBUG | Version matching is enforced. Version: 0.0.3
2016-11-28 19:52:41,544 | DEBUG | content-providers: participating
2016-11-28 19:52:41,545 | INFO | Decisions are completed.

I am now seeing the Tolerance reached message, but it is reporting False.  Any ideas on why?

@cookandy Can you send a longer log of activity via a gist?

hi @kernelpanek-tendril
Here is a longer log... hopefully it's helpful:
https://gist.github.com/cookandy/a03562e41e3eeec11bafe13f2076f0dc

Any ideas @kernelpanek-tendril?

@cookandy I think I found my mistake in one of my prior commits in the history_manager, where I was attempting to remove an early return statement. The history_manager did not have any good unit tests, so the change was not flagged. I've been working on corrections for all of this... a fix, and tests for history_manager.

brilliant, thanks @kernelpanek-tendril!  looking forward to the fixes :)

@kernelpanek-tendril when you say

Your fastscaleup rule:
"mas_rule_fastscaleup_1": "cpu | >10 | PT10S | 3 | PT1M30S",
"mas_rule_fastscaleup_2": "memory | >50 | PT10S | 3 | PT1M30S"

What your rule is set up for is: "If CPU utilization is greater than 10% and memory utilization is greater than 50% for 1 minute, 30 seconds; then scale the application +3 instances." However, your memory usage has not reached the >50 threshold, but your CPU utilization has surpassed the threshold.

how fits PT10S on the sentence

any update @kernelpanek-tendril?

@jmparra The PT10S is the backoff timespan. Once a rule is triggered and acted upon, the same rule should not be acted upon for the duration of the backoff timespan after the last action of the rule. So, when the mas_rule_fastscaleup rule is used to scale the cluster up by 3 at 12:00:00am, it cannot be used again until 12:00:10am.

@cookandy Sorry for the delay. I got terribly sick after re:Invent this year and had a project with an EOCY deadline to finish.  However, I am now on holiday break, so I am picking up where I left off.


installation instructions can be improved for non-python users

I tried doing a TuLiP installation from scratch on a wiped out Mac. I found the following poitns a bit confusing in installation instructions (this could be because of the strange access rights on Mac).

User access rights are not clear in installation instructions. Is it assumed that everything is done in sudo? We might want to clarify this.
I started getting pip and installed it for --user.
pip install tulip did not work after getting pip as setuptools clashes with the one that comes with Mac python distribution.
My attempt to upgrade setuptools with pip also failed due to an EnvironmentError
I finally installed virtualenv and installed tulip in a virtualenv with pip
Then, I kept reading the instructions in https://pypi.org/project/tulip/. To check the dependencies, it says I should call tulip.interfaces.print_env() but it is not clear where I should call this. Maybe we should mention that this can be called from within python.
It also says "A test suite is provided under tests/." but if I use pip for installation, does this folder exist? Maybe we should say, this is for when one installs from the source.

Some of the above points are clearer in the user documentation https://tulip-control.sourceforge.io/doc/install.html where a distinction between PyPi and installing from source is made. Just thought I would write these down in case it is useful for someone else.

while treating this issue, consider switching to Pipenv


Fix infinite recursion and avoid query cache depending on RAND.

This is causing problem in the situation when the query cache
being flushed triggers the initialisation of the RAND subsystem.
The alternative is to use a fast and small xorshift random number generator.
The stochastic flushing doesn't require good random numbers, just enough
variety to avoid causing problems.
Fixes #9455

[ ] documentation is added or updated
[ ] tests are added or updated

A comment in the code explaining why it's not LRU will be helpful to future devs.

Does this comment count:
/*
 * Flush an element from the query cache (perhaps).
 *
 * In order to avoid taking a write lock to keep accurate LRU information or
 * using atomic operations to approximate similar, the procedure used here
 * is to stochastically flush approximately half the cache.  Since generating
 * random numbers is relatively expensive, we produce them in blocks and
 * consume them as we go, saving generated bits between generations of flushes.
 *
 * This procedure isn't ideal, LRU would be better.  However, in normal
 * operation, reaching a full cache would be quite unexpected.  It means
 * that no steady state of algorithm queries has been reached.  I.e. it is most
 * likely an attack of some form.  A suboptimal clearance strategy that doesn't
 * degrade performance of the normal case is preferable to a more refined
 * approach that imposes a performance impact.
 */

I can add more details.
Instead of LRU, least frequently used is also possible.

how'd i miss that? :(  especially since i remember reading it now that you re-posted it.

Your query prompted me to reread it and fix the error, the random generation is very cheap now.
I also added LFU as an alternative.

Ping for a @openssl/committers review.

Merged after removing an EOL space character in addition.
Thanks.

Split into two commits.

Your commits have no subject line? Would you mind adding one (you could just add two linebreaks after the first sentence)?

Ok, maybe I just need some more time to better understand the code...

It's not just the code, the error report took a while to figure out what was going on.
The setup is that the cache has filled with algorithm queries, this triggers a flush.  This uses RAND.
372. is RAND grabbing the public DRBG.
367. is the DRBGs being created.
361. is the DRBG getting the AES-256-CTR algorithm.
358. is the cache flush being triggered because the `needs_flush` flag is still set.
350. is back in the flushing code.
349. is RAND being wanted again.
...
347. is RAND grabbing the public DRBG.
342. is the DRBGs being created.
336. is the DRBG getting the AES-256-CTR algorithm.
333. is the cache flush being triggered because the `needs_flush` flag is still set.
325. is back in the flushing code.
324. is RAND being wanted again.

Moving the needs_flush flag clearing back two lines prevents the recursion.
Everything else is the removal on RAND and some cleaning.

The lock order inversion issue depends on this one being fixed first 😢

Thanks @bernd-edlinger I'll merge to master before @mspncp comments again :grin:
I'll answer any further queries but this needs to be in before the fix for #9454 can be figured out.

Oh, my reply overlapped with the other interactions in a very funny way. That's multithreading, I guess...
Never mind :-)

I'll be honest, the new architectural changes have left me a bit behind too.  I know I've addressed this problem here.  The different order locks are another matter -- one which I am concerned about and which needs to be addressed.
I'll run you through this one over a beer when we meet :)  It's complicated (said few other men with conviction ever).

Yeah, multithreading is one thing I'm actually capable of (trying to) understand.

I'll merge to master before @mspncp comments again 😁

Still choking on that one. The funniest mention I ever received.


[Bug] Downloads only one chapter

If I try to download a novel, it will only download one chapter of it.
The same happens for every novel if I refresh in the Library tab.
The sources that I use are BoxNovel and FreeWebNovel.

Did you use steam instead of download as I downloaded currently and it's fine and yeah maybe you make your app in background so it stop working for fixing that you should allow it unrestricted access.

I tried it with the app in foreground, still the same problem. It also worked perfectly fine with the previous version.

You should try a vpn or use <IP_ADDRESS>

I've tried a VPN, sadly still the same issue. I could try a reinstall, but I would loose all my downloaded novels.

Just copy/cut files folder from android/data/appname and you won't loss anything.

Can't do that, my device isn't rooted.

If solved how about closing this issue!

I just tried to reinstall it, sadly still the same issue.

The previous version 2.1.0 still works fine.
Oh and it seems that the new version does not ask me for a storage permission. Maybe that's the issue

it happens to me too, so i downgrade to version 2.1.0

it happens to me too, so i downgrade to version 2.1.0

Version 2.3.0 fixes the issue and everything works now perfectly.
Thank you very much!


Missing/incomplete module documentation

I created this meta issue to track missing documentation. I added [x] even
thought documentation might not be complete. This point is just to make easier
to people to find what module needs documentation, so they can easier contribute
to documentation.
If you want to contribute to documentation there is snippet template,
M-x -> yas-snippet-insert ->Doom module readme
Or in org-mode you can also write __doom-readme <TAB> to open readme snippet
All these are from develop branch.

[ ] input

[X] chinese (incomplete)
[X] japanese (incomplete)

[ ] completion

[X] Company
[ ] helm
[ ] ido
[X] ivy

[ ] ui

[ ] deft (incomplete, update to new format)
[X] doom
[X] doom-dashboard
[X] doom-quit (incomplete, update to new format)
[ ] fill-column
[X] hl-todo (incomplete, update to new format)
[X] hydra
[ ] indent-guides
[X] modeline (incomplete, there is a lot of TODO)
[X] nav-flash
[X] neotree (incomplete, update to new format)
[X] ophints (incomplete)
[X] popup
[x] pretty-code
[X] tabs (incomplete)
[ ] treemacs
[X] unicode (incomplete, update to new format)
[ ] vc-gutter
[ ] vi-tilde-fringe
[X] window-select
[X] workspaces
[X] zen

[ ] editor

[ ] evil (keybindings need update)
[X] file-templates
[X] fold (incomplete)
[ ] format
[ ] god
[X] lispy
[ ] multiple-cursors
[X] objed (incomplete)
[X] parifier (incomplete)
[ ] rotate-text
[X] snippets (incomplete)
[X] word-wrap

[ ] emacs

[X] dired
[ ] electric
[X] ibuffer (incomplete)
[X] undo (incomplete)
[ ] vc

[ ] term

[ ] eshell
[ ] shell
[ ] term
[x] vterm

[ ] checkers

[ ] syntax
[ ] spell
[X] grammar

[ ] tools

[ ] ansible
[ ] debugger
[x] direnv
[x] docker
[X] editorconfig (incomplete, update to new format)
[X] ein
[X] eval
[ ] gist
[X] lookup
[X] lsp (incomplete)
[ ] macos
[ ] magit (incomplete)
[ ] make
[ ] pass
[ ] pdf
[ ] prodigy
[ ] rgb
[x] terraform
[ ] tmux
[ ] upload
[X] wakatime (removed?)

[ ] lang

[X] agda (incomplete, update to new format)
[ ] assembly (removed?)
[X] cc
[X] clojure (incomplete)
[ ] common-lisp
[X] coq (incomplete, update to new format)
[ ] crystal
[X] csharp (incomplete, update to new format)
[ ] data
[X] dart
[x] elixir
[ ] elm
[ ] emacs-lisp
[X] erlang
[X] ess (incomplete, update to new format)
[X] faust
[X] fsharp
[X] fstar
[X] go
[X] haskell
[ ] hy
[X] idris (incomplete, update to new format)
[X] json (incomplete)
[X] java
[X] javascript
[x] julia
[X] kotlin (incomplete)
[X] latex (incomplete)
[ ] lean
[X] factor
[x] ledger
[ ] lua
[X] markdown
[X] nim
[ ] nix
[X] ocaml
[X] org (incomplete)
[X] perl (incomplete, update to new format)
[X] php
[ ] plantuml
[ ] purescript
[x] python
[ ] qt
[X] racket (incomplete)
[X] rest
[ ] rst
[X] ruby
[X] rust
[ ] scala
[X] sheme
[X] sh
[x] sml (incomplete)
[X] solidify (incomplete, update to new format)
[ ] swift
[ ] terra
[ ] vala (removed?)
[ ] web
[X] yaml (incomplete)

[ ] email

[X] mu4e
[ ] notmuch
[ ] wanderlust

[ ] app

[X] calendar
[X] irc
[ ] rss
[x] twitter
[X] write (removed?)

[ ] config

[ ] literate
[X] default (incomplete, update to new format)

I also created template README.org as gist ([https://gist.github.com/Amatrelan/6e50491992379018f272ca2fc7cdb897](README.org gist) but better wait acceptance to style from @hlissner.

Ah ok! I can use that template to write some module documentations what I use mostly.

@hlissner :lang elixir can be checked off.

@Amatrelan what would be the best way to keep this list up to date?
As of now ui:worspaces, ui:doom, ui:doom-dashboard, and app:write have been updated.

Imo the best way is to add TODOto the headers in https://github.com/hlissner/doom-emacs/blob/develop/modules/README.org and require for that to be updated before any merge.

This could be misleading for new-comers though, since they might see the TODO and FIXME tags as indications that the modules themselves are not complete instead of their documentation.

Well you can always add a small note at the beggining of the doc. There migth be other more pressing issues with this method.

Hi, can I help writing some docs. Have you some style guide or help to write the docs? Do you have any recommendations on where to start?

I recently started doing exactly the same. I started based on this comment https://github.com/hlissner/doom-emacs/issues/1166#issuecomment-462459077
Regarding where to start I guess wherever you feel more comfortable, for instance I started with some refactoring of existing documentation, marked as "update to new format" in the first comment of this issue.
You can also check my recent PRs https://github.com/hlissner/doom-emacs/pulls?utf8=✓&q=is%3Apr+is%3Aclosed+author%3Azakkak although the above comment should be enough to get you started.

@Amatrelan what would be the best way to keep this list up to date?
As of now ui:worspaces, ui:doom, ui:doom-dashboard, and app:write have been updated.

I try to keep up here to update it, but I think currently only @hlissner and me are only ones that can modify original issue.
I added to original issue description how to get yas snippet template to use as barebone for readme files. M-x yas-insert-snippet Doom module readme or just __doom-readme <TAB> Both of those requires snippets module enabled.

@Amatrelan if you create a Readme file in a module that is missing one and snippets are enabled it automatically inserts the snippet.
Additionally, can you please add the following command as an example for figuring out the #+SINCE tag?
git describe --tags $(git log --format="%H" -- $HOME/.emacs.d/modules/lang/ledger | tail -1) | cut -d - -f 1

There seems to be some module structure changes and some were moved to other places so I tried to make more accurate issue. But there seems to be some modules removed so can you @hlissner check those. I didn't remove them but make (removed?) on them so easier to find. And I will check out all trough so I can see if there is some README.orgs put in modules but forgotten to update this issue now too.

I probably should have referenced this issue in the PR itself, but #3060 added a README for tools/rgb

This issue has been mentioned on Doom Emacs Discourse. There might be relevant details there:
https://doomemacs.discourse.group/t/missing-and-incomplete-documentation/156/1


chore(master): release 1.2.61

:robot: I have created a release beep boop
1.2.61 (2023-09-14)
Miscellaneous

deps: update codecov/codecov-action action to v4 (b7bf9b5)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/wayofdev/php-cs-fixer-config/releases/tag/v1.2.61 :sunflower:


frontend: Fall back to browser default clipboard behaviour if clipboard API is unavailable

When the client is loaded over HTTP the clipboard is disabled. In that case, it's probably better to have the clipboard copy fix hook fall back to the default behaviour. In addition, let's have the copy button show a hint that the user should set up HTTPS to get the copy button working.
AI Description

This pull request introduces a new feature to the CopyToClipboardButton and CopyToClipboardIconButton components, ensuring that the clipboard functionality works only over HTTPS connections.
Changes:

The available variable is introduced to check if the Clipboard API is available.
The buttonLabel and label variables are updated to display a message indicating that HTTPS is required for the copy functionality.
The disabled property is modified to disable the copy button when the Clipboard API is not available.
The aria-disabled and disabled attributes are added to the IconButton to ensure it's disabled when the Clipboard API is unavailable.
The useFixCopyBug hook now includes a check for the Clipboard API availability, falling back to the default behavior if it's not available.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 79.92%. Comparing base (7c79633) to head (e4a6a2b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #802   +/-   ##
=======================================
  Coverage   79.92%   79.92%
=======================================
  Files         242      242
  Lines       10305    10305
=======================================
  Hits         8236     8236
  Misses       2069     2069

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.